I will start with a personal note. I am an MA in English Literature and I was teaching English Literature. Now I happened to go to a place which was called at that time Central Institute of English and Foreign Languages. Now it is a flu, it is in Hyderabad where I did a kind of diploma course in teaching of English. In the first semester there were some three courses that I was doing related to language studies. In particular there was a course on Introduction to Linguistics, a course on grammar and a course on phonology and phonetics. So I was doing these courses and I found, I mean let me share this with you, I found the entire subject of linguistics absolutely banal. You know there was nothing of infonology and phonetics, there was hardly anything that was of any interest to me and grammar you know sort of, I mean hardly, I did not particularly care for grammar and during those days there was a negative attitude towards the grammar and all that. So grammar was considered to be something fit enough to be taught at school and not at higher levels. So I carried that you know prejudice with me. So at that time you know I thought I am not, it is okay you know I do not really, I am not bothered about this. When it came to linguistics we were taught things like the difference between human language and animal systems of communication. It was a very traditional kind of topic in the courses of, in introductory courses during those days and that is one topic I found extremely easily. I am not bothered about you know how animals communicate. I am going to go and I will go and teach English to human beings and I am not bothered, why waste my time. And then questions like phonological structure, morphological structure etc. etc. I thought okay I mean these may excite some people but there was absolutely you know nothing that heightened my or that you know made me interested in these subjects. The course was ending and the last 3 or 4 classes were there and my teacher, all the teachers are excellent but the subjects were uninteresting and the last few classes, 3 classes were devoted to a topic called Transformation of the Lama. And I remember in the first lecture, in his first lecture the course instructor said that there is a person called Noob Chomsky and he says that all languages are identical at the level of some underlying representation as a level of you know they share common structural features.  I thought what kind of a thing is this? You know I mean we know that languages are I mean I was in Hyderabad so Telugu was unintelligible to me. The fact of language unintelligibility is you know was nothing new and we know that 2 languages which are mutually unintelligible may not it may not be the case that they have entirely different structures that we knew we had that much understanding of abstraction. We knew that structurally you know the I mean let us say you know Tamil and Odia that is my language you know. So they are unintelligible mutually unintelligible but they share a structure so we knew that that is fine. But then languages which are which do not which cannot be linked in terms of history, genealogy, nothing absolutely unrelated languages okay and then here is a person who is saying that they have an identical structure at some level of at some underlying level. I thought this is something which is marvelous and he is saying such he is saying something which sounds patently absurd and so many people take him seriously. So I think I was in my early 30s then now till I retired from IIT Kanpur those 30 years I devoted to finding out what is all this about okay and I do not regret a moment of it you know so fascinating. I mean see it was it we know that if you say languages are different and languages have different structure each language has its own unique structure you know fine I mean that does not disturb anybody because one would think that okay that is experience you know English has a subject verb object kind of structure my language has subject you know object and verb kind of structure. So there are in many ways languages could differ like this at the level of structure so that is not surprising okay but what is surprising is the fact that there is a kind of common sharing in terms of structures. So what is that so I forgot my PhD research on EM Foster and thought that okay let me see what this is all about I mean it is almost like you know sort of going on a discovery trip sort of finding out what all this is about isn't a kind of you know powerful intellectual experience many many people would have joined you know would have tried to do linguistics for the same reason how come you know sort of how does one account for the fact that languages share that is the claim common features at some underlying level and yet you know they demonstrate different structural properties at other levels okay so that is something which that is what is how does one articulate the underlying similarities between languages and their systematic differences languages do not differ from each other in structural terms in absolutely unpredictable you know ways the ways languages differ from each other in structural terms are predictable right so we know that a language the two languages can differ along certain lines they cannot differ arbitrarily okay it cannot be the case that there would be a language in which take a sentence of you know sort of a take a sentence of a considerable length right and create a sentence which is the mirror image of that sentence and think that if the first sentence is grammatical the mirror image of that sentence is also grammatical nothing of this kind you know so there cannot be languages like this so so you know sort of so this kind this is very fascinating that languages differ surely but they differ in predictable ways in limited ways so you know how what kind of a theory is there to be able to account for all this so that's where you know sort of I think many many people you know I mean I was a small fry but many many people would have been interested in trying to find out what is this all about so that's one and when Chomsky came into the scene you know the kind of grammar that was written was what was called you know descriptive grammar structural you know grammar there was a there was a school of linguistic thought called structural linguists so they are very hardworking and people they did a lot of creative things in their own domain but then when you know they said that you see linguistics is a science they were the first people who said that linguistics is a science and a classificatory science you know it's a classificatory science so what did what did it mean it meant something like this that you let us say you know talking about sentences so let us say we we divide all sentences from the language into various you know sub parts various groups so call a particular kind of category various categories you know call a category simple sentences another category you know simple negatives another category you know sort of simple questions then compound questions you know so categorize sentences and label each category so simple sentences compound sentences complex sentences within simple negatives questions you know imperatives whatever so you know so you collect data very carefully using the most sophisticated instruments of data collection and arrives at sentence level you do lot more you start from the phonological level and then you come to the sentence level and then when you come to the sentence level what you do is you you categorize your sentences in terms of sub categories like simple compound complex just as a tool now see categorizing categorization is not a simple task it's a it's a fairly challenging intellectual task because you are you are looking at the features and carefully trying to say which are the distinctive or distinguishing features of each category so it's not a small thing we should not really you know think that is it's unimportant but when I was teaching at IIT Kanpur I used to say that this is like what happens in Bhattadhukan you go to a Bhattashu shop and you have Panchanambarshu, Sathnamarshu, Artnamarshu in different you know in different you know sort of slots okay now of what great interest is this you know I I mean we trivialized it but now I don't know trivialize you know the categorization is a serious task you know I mean you find out features common features you know distinguishing features so it's it's not unimportant but you see when it comes at the end of it what have you done at the end of it you simply have you know sort of slotted them into different categories and that's about it now Chomsky said no this cannot be a very interesting and serious science you know it can't be interesting and what and science is not concerned about classification although much later I came to learn that today's in today biology is a very sophisticated subject but only 50 years ago biology was a classificatory science I mean about 50 years ago okay it was more or less a classificatory science okay so maybe disciplines have to undergo a stage of classification you know maybe I mean questions of explanatory nature maybe it's difficult that such questions are asked at a certain level of an intellectual enterprise so maybe a kind of you know a kind of stage has to be gone through where classification becomes extremely important but then Chomsky said no we need explanatory theories okay so what kind of explanatory theories we need what do we explain okay so you know for the first time I mean he not only he said what kind of a science linguistics would be what kind of a science is this you know at that time today it's slightly different but at that time there are no experiments you know you couldn't conduct experiments okay right you know you you come up with you postulate linguistic rules but there are no experimental that there are no experimental evidence for the same right so some people you know who wrongly associate science with explanation sorry associate science with experimentation you know I mean mature sciences like physics and all we know that I mean sort of you know the how important experimentation is there for those sciences so because of the success of the natural sciences so sometimes you know people tend to associate experimentation with scientific inquiry so if your science or what you are claiming to a science has no experiments right you know your science cannot be taken seriously it is at best a kind of you know folklore or it's at best a kind of you know pre-scientific science or whatever if you call it a science okay right so don't make any scientific claims for that you know so you don't have an experiment so how does one get support I mean for the postulations that you are making okay so that was there but then that's not a very serious I mean I you know see for example in astronomy okay what kind of experimentation do you have in astronomy and is astronomy art I wish it were but you know is it art you know it's not so it is not the case that in all sciences you know you would like to have experimentation now if you don't have experimentation in your you know in I mean in in your domain of inquiry which you want to call science okay how how does one evaluate the findings right how does one evaluate the findings now whatever findings you have arrived at how do one check now if there is no way to check whether your findings are correct or incorrect or whatever right then we are not doing any you know we are not into any rational enterprise we are into some kind of mystic enterprise okay you say something and by definition or definitionally that's true you know it's almost making a claim like that so what so so you know so what does one do so all right so you write a grammar of English let's imagine you write a grammar of English okay now so what do you do how do you test it okay the simplest thing is so long as you are writing a grammar of English you test it against the data so does it account for all data okay does it you know account for in this case will mean does it cover all data right so if it covers all data you think that okay maybe you know there is I mean it's it's right okay but soon you know that I mean you are not you know an enterprise cannot stop with writing the grammar of or working out the grammar of a single language because English is a language so is Hindi so is Odia so is Swahili so are many other languages right they are all objects of the same type you know they are all objects of the same type right so so a kind of descriptive mechanism or a descriptive device that you have arrived at for one language right should apply in you know two other languages as well because there is no science where for each object of inquiry there is a there is a kind of theory so and it's not like the it's not like the it's not it's so it cannot be the case that for Hindi there will be a theory of grammar for English there will be a theory of grammar and these theories of grammar will have no there will be no way to converse I mean no conversation would be possible between these theories of grammar that's a false thing it was noted way back in 57 it was noted that all grammars of languages you know I mean people wrote grammars of various languages now then you had notions like subject your notions like object your notions like noun verb pronoun like that verb you know right so is it accidental that when somebody writes a grammar of English he uses these terms you write a grammar of Sanskrit you also use these terms you know right so is it accidental that languages when you write grammars of different languages you know the grammatical vocabulary is something which is largely similar how does it happen okay huh so the question that arises is where do you get your basic terms from okay right where do you get the terms from so these have to come from some kind of a theory there has to be a theory of grammar and from that that theory of grammar would itself specify that these are the kinds of notions that we were going to have and these are the kind of interactions that we are going to have between you know whatever entities so what are the entities how do the entities and you know sort of interact so these are things which cannot be just for one language they have to be for all languages okay it's not an accident that you know you have I mean as I said you know sort of it's not an accident that subject object verb you know noun pronoun etc categories category labels whenever you write a grammar for any language these are the categories which you use how come okay huh so if you have to find an explanation for this you cannot find an explanation from history of writing of grammars no you can't say oh let's write a history of grammars of all languages you know and in somewhere you won't get it anywhere you have to have something like a linguistic theory a grammatical theory and the theory will give us the terminology the basic concepts the interactions everything right so for the first time in the history of linguistics in the linguistics there was felt the need for a theory of language or a theory of grammar see linguistics is not you know I mean linguistics are not always called linguistics it was called grammar or whatever else okay but linguistics is a civilizational subject you know it is not something which is recent 2500 years ago in Greek civilization you know sort of linguistics was being done and even earlier in India people are doing linguistics we talk about Panini who is probably 2500 years old but Panini is only the finest flowering of a tradition that existed Panini you know it's not that grammatical tradition started with Panini no lots of people had worked before Panini worked so this rich traditions you know so grammatical studies are that old okay but for the first time in the history of grammatical studies you know the need for a general language the general grammatical theory or general linguistic theory was felt I am not saying that people who wrote grammars before did not have a theory of grammar they had a theory of grammar but what that theory of grammar was never it was never explicated because it was not necessary to spell out the kind of grammar that they had okay kind of theory of grammar that they had in other words you know what is the theory of grammar what are the goals of study of grammar right what does it explain something does it categorize something what does it do okay what are the upper boundaries and lower boundaries you know the usual question that you ask about theories right right so you know and what is architecture you know they had some notion surely if you don't have these theoretical notions you cannot write a grammar right you know the person who wrote a grammar of you know Sanskrit phonology morphology etc etc or Karaka had an idea of what a grammatical system is like what a language is like what kind of structural you know system the language is likely to have that is why he divided them into phonology morphology whatever he had but then there was no articulation of it there is no explicit articulation of it so the need for an explicit articulation of the theory of grammar felt so strongly and you know was realized and you know it I mean this is I mean when Chomsky came into the scene so we must remember Chomsky as a man right you know who for the first time articulated the the the need for an explicit linguistic or grammatical theory okay and then you know the the other thing so you know so you have a theory and all these will follow the grammatical terminology will follow and terminally just you know I mean one part of it because you are talking about what is that what are terms the terms are terms are labels of entities and next is interaction between entities and you know they they interact within a framework and that's the architecture so all these things would come okay at at at later stage all these things would come okay and then you write your grammar in terms of that right now suppose you have postulated something right in your model and you are saying that you know this is something which is a general feature of all languages imagine something like you know what I am going to say okay imagine something like you have done phrase structure rules right you are familiar with phrase structure rules okay imagine that what that that phrase structure rules are part of universal grammar a term with which you are familiar right let us imagine that phrase structure rules are a part of universal grammar now how do you know the coming back to the question as to if we don't have experiments right how do we test you know our findings our hypothesis okay right so how do we test so it is like this if you can arrive at a grammar where you can dispense with phrase structure rules completely right you know that you would say that that is something you know that's a that that's a test so when you you know you you can postulate a theory of grammatical universals right but then this theory of grammatical universals whether you know a particular rule or a particular mechanism or a particular you know whatever right kind of rule etc huh so if I mean this rule has to has to stand the test of scrutiny you know from other languages right so you write a grammar of another let's say there is a you know let's say you are postulating some some rule on the basis of English let us say that you can do you can say that we have phrase structure rules in English grammar and we our hypothesis is that phrase structure rules do not simply belong to the grammar of English phrase structure rules belong to you know they are a part of universal grammar now if somebody works in on I mean if somebody is working on some other language and is able to argue that you don't have a need for the phrase structure rules that you have postulated and you are claiming universality or universal status for them right then the phrase structure rules are not sort of I mean they are question they cannot be so the test so the testing device come from careful analysis of another language of the grammatical you know features of another grammatical systems of another language okay huh so please notice one thing you know you can't you I mean data cannot falsify a theoretical claim right suppose you know you are making a theoretical claim about further phrase about the fact that phrase structure rules exist as part of universal grammar right they are a part of universal grammar okay so let's imagine that this is a theoretical claim that you are making at the moment it is an assumption you have posited it on the basis of study of a single language a careful study of a single language okay right the testing comes not from the fact that there are sentences in other languages right which you know seem to falsify the claim say data cannot falsify the theoretical claim right data cannot direct on analyzed raw data cannot falsify a theoretical claim okay what can is analyze data into a system of rules a rule I mean if there is a rule that exists or you are claiming a rule to universal a counter example will be from you know from the grammar from another grammar in terms of another rule rules you know that kind of thing you cannot you cannot falsify you know direct on analyzed data cannot be used to falsify you know or question or undervalue you know theoretical postulations in this particular case as something like a theory of rule okay so you know you know what move alpha okay right know what move alpha okay if you find an example from another language which says that we really don't need something like move alpha okay we can do whatever we have what whatever move alpha does we can do by using some other device and we can do something more and if you know so look at the level of claim the claim is being made at the level of a rule not at the level of you know I mean direct raw data okay right so so what Chomsky did was you know it's interesting what he did was he not only created a sort of field in the sense he raised new questions with respect to grammar he also provided a way of how to test them because experimental testing at that time was not there okay okay now today there are experiments you know people do conduct experiments right but you know I am not going to get into that and there are all kinds of questions that can be raised with respect to those experiments as if you want to use them to falsify the original there are complications there you know they are not straightforward so let's not go into that okay so all that I am so far what I have tried to suggest is this that linguistics was considered to be a classificatory science when Chomsky came into the scene Chomsky said classificatory science is okay but not sufficient a discipline has to be you know it has to be explanatory in nature so you have to come up with explanations now what kind of explanation what is that you are explaining okay data the data or what you are explaining you know right and and and what is what is explanation you come up with more and more you know simpler more elegant grammars so the grammar itself is a kind of explanation you know for the kind for the data so you know it's explained by grammar okay and the falsification of this or testing of this will come from grammars of other languages right okay so something like that you know so roughly sort of so this is what it is and many many interesting questions started getting raised say for example you know consider something like subject object and verb right you know take them as three different categories so how many combinations are possible eight combinations I think you know you can permute them and combine them in eight ways right so you know so what would we expect we would expect that we will encounter languages in the world right which will have which will which will demonstrate each one of these categories right so there will be no languages which will demonstrate each of these categories but the fact is that it's not really like this you know most of the languages are SOV Hindi type SOV languages you know I'm quite a large number of languages are English type you know is VO some are VSO okay and maybe there are certain other types okay and I'm forgetting details but there are certain categories we do not show up at all okay so so that's the kind of thing so you see if you have a certain kind of you know I mean if you have this VSO right you know I'm sorry if you take this terms you know subject verb and object right okay and think in term and you know sort of think of languages right so you will expect something like eight categories and you will expect the world to demonstrate this category but that doesn't happen so what we are dealing with here is not a question of logic but something else logically you would expect all these you know sort of categories of languages to be there in the world but they are not there so what we are dealing with is not really a question of logic but something else okay now why is it that most of the languages are of you know SOV and SVO types and there are languages language types which are not simply there why is why is the world like this so there is a variety you know linguistic world you know there is a variety it's not you know it's not uniform in the sense that there is not just one type of language SOV no there is a variety okay but the variety is a constrained kind you know it is not that whatever is you know sort of whatever but the all the logical possibilities are realized in actual you know in actual reality of languages it's not like that okay so so this is a question to ask why is it like this now look at the kind of question that is being asked okay right these were never these were never the kind of questions were asked before Chomsky came into the scene you know right in the in the 60s and all these questions are early 70s questions were getting asked and these questions in all those thousands of years of our you know rich linguistic you know history no one ever asked these questions you know the reasons are of course simple I mean you always looked at one language so you wrote the most economical grammar of one language right you never thought that okay you know here is another language here is another language you know right okay so what I am saying for this language and if I find that there are similarities you know I find in other languages which I am exposed to in day to day life okay it never you know it never attracted the attention of anyone of them to find out is this similarity accidental okay it is not that you know sort of in our I mean panemi or our great Sanskrit grammarians did not know any other language or not exposed to any other language they were you know but you see there are certain there are certain attitudes because of his never cared to study them you know if you look up I mean the tradition I am in a sort of Bimal Mathilal says that our Sanskrit scholars you know grammarians would study the language of somebody of a category of people whom he called sister sister sister sister is one who is not just a scholar but also a man with you know spiritual attainments see so if you say that we are going to study the language of a set of people okay and this and this will be kind of you know elite okay and we will not bother about the study of any other you know sort of groups of people right what would you do this is one of the consequences that even though you find that you know the people who you have defeated in the defeated in some war and taken them as slaves to your households okay they speak a different language you will not treat that their language seriously consider the language seriously because you do not treat them seriously and you have decided that you would study the language of only you know I mean in some sense you know the purest kind of language okay all these terms make no sense in the democratic world okay right but you know if you decide to study that that is to everywhere in the western tradition you know look at the grammar books the best of writings of the best of authors were what where the data for grammatical analysis spoken language of hardly analyzed debased corrupt you know anyway sort of so so you see because I think you see it is very difficult to find a social I mean to to to you see if you are writing I you are looking at the history of knowledge it is very difficult to explain certain things in that history in terms of sociological facts you know I am not saying that they should explain that but perhaps you know if we did not you see all these things on I mean if we did not in our civilization consider the fact that if Sanskrit has a structure like this you know some other language has also similar structures even when you notice you know you did not raise any question out of it you know right and that is because we were possibly you know sort of we did not think that languages of others was considered to be a serious subject of investigation and sister so long as the sister attitude is there we will study the language of the sister the loan and sister you know is defined not as somebody who is a scholar but also as somebody who has spiritual treatments you know who is studying for the sake of the study who is studying for you know for not for any benefit but for general good you know things of this kind so if you do that you are your data domain is is shrunk okay so other similarities and dissimilarities you know you just ignore right it is only when you discover it is only when you note the similarities and dissimilarities that your mind gets challenged to find out how come you know there are how come there is a similarity and how come there are dissimilarities so you will try and explain this in terms of you know I mean in some term okay one is you know postulate something called a universal grammar and postulate parameters you know which will define you know the the systematic differences between grammars okay so you know only when you only when you notice this and when you only when you think that this is something that is worthy enough for investigation okay so languages the some features are similar some features are dissimilar how does one explain dissimilarity and similarity right how does one explain dissimilarity and similarity right so you can do it in many ways you know I mean you can do it let us try and see what we can do with with with this simple observation that you know let us say given two languages or three languages we find certain similarities between them at the structural level right and we find certain dissimilarities how do we go about it so suppose we say you know the fundamental principle all languages differ right structurally right you know suppose you saw all languages differ structurally right period you can't raise the question about similarity you know if languages differ from each other in you know and that is the story right okay that is what your observation is that languages differ from each other right so suppose you say we have a theory and we look at the grammars like this all languages are you know each language is unique and is describable in its own terms you know so suppose you have this kind of a theory okay right the question of accounting for similarities cannot even be raised right now suppose you have a you have an alternative theory and says all languages you know I mean there is a lot of see you finding similarities we say all languages are similar and you know suppose you say this you know it is false it just cannot be right so then but if you take that kind of a view you at least have a possibility of trying to find out okay given the similarities how do I account for the differences okay and when you discover the differences are not you know I mean sort of wild and unpredictable right you know arbitrary then you will have a way of trying to find out how to you know sort of account for this similarity and dissimilarity right okay so that is universal grammar and that is parameters okay right parameters where you know sort of in one model they are called the sort of the term parameter is used for them in one model earlier the term was not used something else was used but essentially the fact is how do we account for the similarities between you know grammars of different languages and their systematic dissimilarities so you are working out in various models at various times in many ways okay huh so this is something okay now let us go to something else see in all cultures you know another two o'clock yeah in in in in our culture and I am sure in other cultures as well at least in our culture see look at you had people like you know we had a rich grammatical tradition you know Panini and all right so you ask them question okay how is what is the you know I mean what is the grammar of Sanskrit right one one extremely good answer is Panini's grammar an economical detailed economical sophisticated formal generative grammar Panini is the first generative linguist and not Chomsky because you look at Panini's funnish rules they they are generative rules okay so in any case you know that is another topic in any case also so I mean very sophisticated grammar right so what is the structure of Sanskrit Panini's grammar is the answer huh what is the structure of Sanskrit Panini's grammar is the answer how is Sanskrit learnt or how is you know how is Sanskrit learnt in Astadhyayi you will have no insights about how this is all learnt okay right so those who really worked out the structures did not address themselves to themselves to the question of how these structures are acquired by people or learned by people no right similarly you know at a later stage there are thinkers in our culture who raise this question how do children learn a language okay so and they came up with answers you know like see through observation you know some I mean they find that somebody is telling somebody bring the horse so actually somebody goes and brings a horse okay right and somebody to pass the salt so you know so I mean you they listen to the sentences through observation they make inferences and all that so you know they learn from experience they learn from explicit teaching they learn from the grammar books they learn from you know lexicon the dictionary so many ways you know right okay now what they said would resemble very much what our behaviorists said about learn you know the the behaviorists you know positivists you know what they said about how things are learned essentially things are learned from experience okay huh so nothing like my mind is a clean slate you know I mean I am overseeing everything really okay so the mind is a clean slate so there is nothing in it so from experience we keep on and sort of we abstract from experience and keep on adding knowledge right so that is how it is it is done so so somebody says you know bring the horse or sort of whatever okay right bring the horse or bring the food and you find that this is happening so from this the child infers the the the the drama and it is supplemented by classroom teaching grammar books blah blah blah and all these kinds of things okay so essentially this is a kind of behaviorists account essentially it is a behaviorist account of you know language learning okay right and these people who were concerned with the question of how a language is learned okay did not ask themselves the question what is the structure of a language so my the point that I am trying to make is that you see what is the structure of language was a question which was independent of the question how is that language is learned or how are those structures learned so the structure of a language and the learning of a language were two distinct questions okay huh and they were answered in in different ways right now what Chomsky did was in some sense he combined these and that is a great achievement okay huh so you know he said you know questions like this okay how do you compare you know in the beginning right when he came up with this you know generative grammar transformational generative grammar and there were the structural grammars before that you know structural grammars traditional grammars before that all those grammars existed so the so the natural the question arise how do you compare these grammars okay huh so you can say no you know sort of I mean generative grammars try to show sentence relationships you know that what is the relation between active and passive what is the relation between you know an imperative and a non imperative you know like what is the relation between sentences right so you apply a passive transformation or whatever and you get a passive sentence imagine you know that let us imagine this okay so you have an active structure apply a passive transformation so you get a passive sentence okay just one thing you know so transformations are not applied to sentences they are applied to structures okay in any case so you know you are trying to find relationships right okay that is coming is a fact it is a fact that is coming so you relate these sentences okay huh right so let us say generative grammar was trying to find sentence relationships and account for sentence relationships what what does that mean explain or account for in terms of the grammatical system the system would say that okay you know we can generate these sentences and all that in right okay now structural linguists would say no we never tried this you know we did not try this so why are you comparing us with generative linguists and say that we are inferior in some sense okay because you know we never tried this you have changed the goalposts so right okay so you you can change the goalpost and score goals over us you know because we never you know what we do we never did then how do you compare grammars then now the answer that was given at that time was you know that grammar is the best or that grammar is the most appropriate the most suitable grammar which explains language acquisition facts better right so look at the connection now huh so you know sort of your there is a structural grammar and there is a generative grammar generative grammar is trying to account for sentence differences who is sorry sentence you know relatedness whose structural linguistic did not try to do so the question is these are incommensurable the goals are incommensurable we are trying to do x they are trying to do y and you can't compare you know these two models to these two models to grammar because you know their their objectives are different how can I do that okay then the answer to this is yeah you still can compare them on a different parameter and that parameter has nothing to the grammar that parameter has to do with you know how I mean which explains language acquisition facts better right look at the assumptions you are making now the assumption the assumptions you are making is I mean if if if if if if if a if an architecture of structure right can be evaluated in terms of how it is acquired right you know I mean what are you doing you are saying that you see we are acquiring that architecture because the mind is ready in some sense the mind is you know predisposed in some sense to learn that architecture or to acquire that architecture okay huh right you know if you are able to learn a system right you know right you learn that system because you already have some means in your mind some equipment in your mind in terms of which you learn that system okay huh so now the system and look at the relatedness between the system and cognizing that system or you know or acquiring that system you acquire the system right because there is some system in your mind which really you know in terms of which you acquire that system more easily than some other system right okay huh that is what you know that that makes you feel that okay maybe you know there is a genetic environment we are born with a certain capacity right we are born with a certain kind of you know I mean sort of gift nature's gift in terms of which we make sense out of our linguistic experience what does that mean to make sense of linguistic experience you know we are able to construct a grammar from the data that we are exposed to right as children right you know we are able to do that because there is already something in our mind right in terms of which we do this okay huh okay you know otherwise we can't right you cannot make you can make sense out of experience only in terms of an intellectual you know only in terms of knowledge without knowledge you know there is no way you can understand experience right this is a very very important thing that you know that we have to keep in mind okay huh okay so you know so I mean so how do I get that where do I get it from it cannot be taught okay you can't teach a one year old child you know right a grammar right you know what kind of instruction do they get they get some simple instruction about this is the object this is called this is called that is the mother this father no etc etc okay very simple but then no sophisticated information and yet a whole grammar you know a system is learned so that it simply means that by you know we are somehow by nature we are gifted with some kind of a knowledge right in terms of which we understand that we make sense of experience and acquire that knowledge okay call that knowledge now you know so I mean only saying this is not sufficient okay so what is that knowledge how do I describe it now call that knowledge universal grammar so what am I born with I am born with universal grammar right now another name for universal grammar is the initial state of knowledge of language okay the initial state of knowledge of language right what is and how do I describe initial state of knowledge of language by in by working out of the architecture of universal grammar now you see the connections now right you know which grammar is the best out of a number of possible grammars that grammar which explains language acquisition facts better right now what does it know I mean what does it really mean that you see in order to acquire language in order to acquire a language or the structure of a language we already have to you know sorry we have we have to have a structure in terms of which make sense of other structures explicitly available to us okay huh so now look at the connection and I think it is a supreme creative it is a you know it is a it is a it is a demonstration of the supremely creative mind of Chomsky which connected these questions he both of these are civilizational questions how do we learn a language okay look no civilization asks this question how do we learn geometry okay you know but it is asked you know I mean obviously the reason is very say you know I mean not a we know why because you know geometry is always I will learn at a later level okay but language is learned you know at by one year one year olds you know demonstrate their understand their command of language okay that does not happen in case of mathematics right so obviously you know you do not ask a question like you know how do you know Artha Shastra right okay you learn it you know to express it instruction but in language case it is different right so anyway so this question as to how a language is learned has been asked you know I mean by people in various cultures thousands of years ago how do I describe the structure of a language is also a question which had asked even probably before you know these acquisition questions were asked okay huh they were different questions in Panini Astadhyayi you will have absolutely no account of how this is learned in Kumarila Bhatta and other you know philosophers speculations who raise this question of how a language is learned you will find absolutely nothing about the phonological structure of the morphological structure of a language okay they are different questions now see these different questions are you know sort of seen as one by Chomsky okay huh so universal grammar can also be described as the initial state of knowledge of language okay huh and within that initial state of knowledge of language you know is what we are born with okay and I will conclude because there is a lot to say but I will conclude by making only one point see why is this enterprise so exciting this enterprise is so exciting you know because hundreds and hundreds of unresolved questions okay huh so you all the time feel that you know what we know is probably not right okay right maybe you know there is there you know I mean see we have some confidence okay but we are never sure that we have really we are even approaching we are you know going the right way we do not even know one simple you know problem is this see you know that I can I mean one can analyze the structure of a sentence without knowing the meaning of that sentence colorless green idea slay furiously you know you can you can assign a structure to this without you know bothering about this meaning okay so you know I mean NPVP whatever you know right you can you can work out the internal structure I have tested the following you know but I but you know I mean my testing coming wrong but you know I have I have done the following I have you know take it to the level of discourse a discourse comprises several sentences right you know 10 sentences let us say you know so I gave a paragraph I mean I gave some three you know I mean pays to my students okay and it was on the football or Pele okay you know a beautiful writer by Edward Gary I know so I just removed the paragraph for divisions and all that and gave the text to my students one would think that you know if you are talking about structure of a written text one would think that chunks you know you will you will divide them into chunks and in some sense a chunk would correspond to a paragraph in print okay suppose you know you do that so that is an assumption which I had what is the structure of it of the discourse you know right so we can think okay you know we can divide them into chunks which is what the computational language some of them who are working on discourse do and then you know I mean so let us say we do this okay now we give it to students right see you would find you know this is what I found you would also find that people divide them into paragraphs depending on the meanings because if they do not know the meaning they simply would not be able to arrive at any decision with respect to which which you know become a chunk and which not how does it and how is it if it is correct that the structure of a sentence can be analyzed or can be arrived at without reference to meaning at all you know that is absolutely clearly demonstrated right you know okay and you know whereas in course of discourse which is actually some total of sentences this simply does not seem to work then what is happening you know can we have one theory for sentences another theory for discourse which itself is you know comprises several sentences no I do not know what is happening you know so you see there will be there are 100 and there are much more technical internal questions right which we have and we do not know the answers to those questions right we are working on them you know we do not know the answers so an enterprise is alive and kicking if there are unresolved problems okay huh right so that is why you know I mean this program or doing generative linguistics is exciting I will end by making this one only one observation about the contribution of Chomsky see there are many people who have at many times said that Chomsky is has misled linguistic community by you know his theories okay right but he has done something you know and I would I would even say that maybe all that we know today principles and the parameters theory you know whatever all that we know today may turn out to be absolutely blatantly wrong possible you know maybe more interesting more formal grammars would arise and you know would I mean they would they would emerge and say that all this is wrong but there is one thing that you know cannot be proved to be wrong Chomsky has said the agenda of how to do linguistics okay right these formal mechanisms may be substituted but nobody can say that is you will have one theory for Sanskrit one theory for you know one grammatical theory for Sanskrit and one grammatical theory for you know Hindi you can't do that you have to you know tackle the question of universals you cannot go back to an earlier stage and say each language has its own structure no that is gone so you have to tackle the question of universals you cannot anymore say that when we talk about the structure of languages we will not raise these fundamental questions regarding acquisition no you know so whatever theories you come up with okay whatever you know new balances you come up with where you know what is a biologically given and what is learnt from experience there's the same thing not sure and nature and culture what is given what is learnt through experience so there may be many ways of you know fine tuning bit between the relations relations between these but the fact remains that there cannot be a day there will not be a day when you would say that okay we need separate grammatical theories for this for separate for different grammatical theories for different languages we will not ask acquisition questions when we raise questions of structure these are not the things you know that I mean we cannot go back to those days of innocence and that is you know I mean here is a man who has told us how to do linguistics so the technical solutions may all turn out to be wrong but there is no going back to universal grammar okay there may be different articulation of the grammar the architecture that we may come up with may not be the same but you know there's no question of saying that no it's wrong I mean that we don't want to do it anymore I think I'll stop here thank you


Welcome back folks, Dow. Dow is up 175, Nasricks up 48, S&P's are up 22. Let's go over to our man Mr. Teddy Kakes. We do each and every Wednesday at 40 past the hour. You can reach Teddy folks at every trading day at 4X-trading-unlocked.com. That's 4X-trading-unlocked.com. In fact, when Tommy and I were hunting in Teddy this morning, you go up in the site and they're pushing us right to the St. Peter's Power and Sailboat Show, which we actually can see out our windows. Goodwill of geolocating ads online, right? There we go. St. Peter's Power and Sailboat Show, let alone the great site 4X-trading-unlocked.com. There we go. How's that, Teddy? We can look out our window and we can see the boat show. It's right outside. We're only a couple of blocks away from the water. But that was so cool, man. That thing to be looking at is pretty cold here in Chicago. Oh, yeah. Well, hey, we get a lot to talk about today, man. We have a very good day to talk about the currency. The dollar index is making some pretty big moves over the past couple of days. Yes. Yes. And how about that pound? It looks like the pound just broke topside, right? We finally got our breakout to the upside. And I was looking for it to hit 131 after maybe a couple of sessions, but it just did it in one little shot today. So very impressive move. So when you get something like that, I guess the pound, we get a couple of weeks for knowing what's going on over there. But when you get a continuation move like that, are you looking for like a 133, 135 in the future? Or does it just stall out for a bit? How does that normally work? Well, normally, if it wasn't because I'm looking at the dollar index right now, I would say that I think the pound would probably take a little bit of a slower grind up towards $1.32, $1.33. Okay. But if you look at the dollar index in the past couple of sessions, it made a higher move high. And then as of yesterday, it made a lower move low and had a really big range over a couple of sessions. That's a very negative indication that the dollar is probably going to be under pressure for a couple of days. And I think that might give a little more lift to the pound and have a little more volatility. I mean, the pound, when it starts to move, if a day like today could be followed up by another day that actually moves at two full dollars easily. Really interesting. And it's been down for so long, right? I mean, depending on how long you've basically been around folks, the pound, you don't see the pound a lot of the dollar 30. Every time you go to Europe, well, every time you go to London. Not recently at least, yeah. The bottom line is that I remember at $1.90, I remember at $2.10, I mean, it's like a closet over there would cost you a fortune. Brexit's been going on for three years plus, so it's become a new norm, yeah. Right.  I mean, it is held solid with very tight ranges, which is not normal for the pound. And I think that with their election, I think it was December 13th or 16th. I can't remember. It's one of the states. So I was expecting it to kind of just grind higher if, you know, things were looking kind of positive for like, you know, Boris Johnson. But now I think with the dollar index under pressure like it is. And if you look at the Swiss, it looks the same as the dollar index. It made a higher move high. It looked like it wanted to really kind of hold that parity. And then all of a sudden it collapsed, you know, and made a lower move low. So that's a neutral to bearer sign for the dollar index, the Swiss. The yen, however, that's the one I tell you guys, it's amazing how it gets beat up with the dollar initially. And then all of a sudden as the dollar is getting pounded, somehow the dollar finds strength against the yen. Yeah, yeah, which I don't want. And it is what it is. I get that. There's no doubt about that, man. It's the one anomaly of the major currency cross rates. I mean, the New Zealand dollar looks like the British pound to today. I mean, it's unbelievable how high it is. So and that's a big deal because when you start to get your lesser majors that actually are moving that much, that volatility, that kind of a range, then no matter what, the dollar index is going to look heavy. And then currencies like the Euro, which were up today, you know, are kind of almost unchanged, you know, it's rebound and make a run for the highs again at the pound stays up there. Yes, no, I can see that. And you know, it's intriguing to talking about the currencies from, you know, other smaller nations, but a lot of them are commodity countries. That hits the commodity market pretty intense.  In the context, these countries have been used to low currency rates and, you know, the commodity market itself, I mean, in the long run, they're going to be 10 times better. You know, of course, if the commodities go up, because then they're all going to have strong currencies, you know, the very West CRB index. Now, yes. Yeah, the variation, man, like in some of these, I guess in all countries, I mean, I listen, I dollars that's 70, what, you know, eight years ago, that's 98. I mean, huge moves. I guess it's, you know, all currencies, right? I mean, they move quite a bit once they stop moving. They trend. Yeah, they trend. I know. Momentum. So but I think the dollar index is something to key off of. And now these rumblings, you know, it's a trade deal stuff is kind of always scares me because I know a lot of stuff overnight, especially in Europe, they were saying it looks like a US trade deal is starting to look closer to possibly happening. That really scares me when that starts to make global news because tomorrow is the day where all of a sudden we'll say, oh, the deal's off the table again or something. You know what I mean? No, that was yesterday, Teddy. There's those women. Yes, I agree. It's that it's that dramatic. My tagline is what a difference a day makes even this morning doing the updates because it really is remarkable that yesterday it was hold on where we got a full 11 months until the election 2020. And then this morning it was wait, it's going to happen before December 15th, 11 days, 11 months, 11 days. That's correct. Yeah. It's amazing how those things swing like that, right? Yeah, well, you know, you can see in this one fundamentally why it would happen.  Yeah. If you're bringing stuff into the country, and those taxes go away, it makes a big difference. Now, if those taxes don't go away on the 15th of December, there's going to be more taxes that I got to put in. And my understanding is that most of that on the December 15th is on consumer consumer goods. Yeah, you're talking about clothing, retail, $160 billion worth of goods. And that's coming into an election here to bring it in again. You know, is Trump really going to hit consumer goods as election season comes in? Why he's going to put a new bill in that all consumers are going to get money like all the farmers got money. Oh, boy, here we go. Right. About now, we do have the impeachment hearing that started today, too. So I mean, I would be that's something that could weigh on the dollar as well for a couple of days. Okay. And then I guess you get year in funding, we get the repo market, the repo market folks would happen is that the Fed, you know, they're oversubscribed by like the billions. They put on 15 billion first, 48 billion came in. At the end of yesterday, they went up to 25 billion. So that repo market is the banks don't need more money for the end of the year. I'm surprised the Fed didn't go right to 48. It's like, okay, there's still shot 25 billion. Yeah. And we're December 4. So, you know, that it'd be interesting to see where that scramble comes from. You know, because I believe that the US banks need dollars, you know, I mean, say they need money, but they need dollars too. Yeah, you know, you can't you can't basically hold it. I believe in other currencies. As you commit to that close of the year. No, no, dollars count on the balance sheet. Yeah. Not not not not yen or Frank's or whatever you want to put put them in dollars. Yeah, the rebalance the rebalancing is coming no matter what. Yeah. Yeah, that's a lot coming down the line. That's a good quote. We're gonna have a wild holiday market this year, guys. I like it. I like it. Listen, folks, you can reach Teddy every trading day at 4x dash trading dash unlock.com is for us. That's trading dash unlock.com. Teddy, you have a great one. Safe one. Stay one. We look forward to speaking the next Wednesday. Thanks, guys. Thanks. Lower the dollars next week. Oh, I like it, man. Seven days. Thanks, Teddy. Stay right there, folks. Tommy and I come right back.


Glad to see you guys sitting here at this hour on Friday. So and also privilege to conclude this great event and the opportunity to conclude this event. So today I'm going to cover how are we finding Dilkidil at Snapdeal for you guys. So for me, I'm Gagandip Juneja having around 70 of experience into software development, working as a lead software engineer with Snapdeal and also working as a PMC member to Apache Blur and Apache Blur incubating. So this is how our agenda session would look like. Introduction, collaborative filtering, a brief discussion, capturing data or solution, technical challenges, what next? And definitely I don't let the bell run. So what are the recommendations? So from for any e-commerce portal like Snapdeal, so for us recommendations are like predicting a new product or new items a user would like to buy or like to see on our website. So what kind of problem that we are solving by doing this? Like personalized user experience, let's say if you're coming on our website with something in your mind and you will see directly on the homepage of the site itself in the recommendation section, so how great your experience would be. And obviously from company standpoint, increasing conversion rate. So we already have the talk on collaborative filtering but anybody don't have idea what the collaborative filtering is? Okay, one or two hands. So okay, we can quickly cover this. So collaborative filtering is the idea where we try to find the, we try to predict items for user based on the other users, which we predict somehow that they are likely, they are having similar kind of interest with the user. So let's say in the diagram, I hope you will able to see, you are able to see it. So from the diagram what we can see is we are having three users, user one, user two and user three and we are having four products. One is grape, second is strawberry, third is watermelon and fourth is oranges. So in our database what we have is we have the history that user one has purchased grapes, watermelons and oranges and user two has purchased grapes and watermelon. So based on this history what we can infer is that grapes goes very well with watermelon and if any third user comes to our website and we know that the user has already purchased the watermelon, then we can easily predict him on the grapes based on the historical data that we have. So if you see from the mathematics point of view, so we just need to create one metrics, item to item metrics, which represents how many times item one and item two have been sold together or bought together. So you can see that item one and item three is here, the cell of item one and item three is having score two, so it's been sold twice together. And if you want to predict for user three, but our recommendation, we just need to take a dot product with the user three purchase history and we can find in the green cell you can see that we can recommend item one to the user. So this is a very brief and basic idea how collaborative filtering works, a lot more details into it, but this is how it works. And yeah, so now we are going further, further in the session we are going to discuss a lot about data, like how we are generating recommendations and why do we want to do that and what are the new tricks or tries that we are doing with the data. So for that first let us understand, let us take one example of our user browsing history snapshot so that we can refer it to for our upcoming algorithms. So this is the snapshot of a user history with Snapdeal. So I can summarize this data for you, from this data you can see that the user has visited nine days ago, he's visited mosquito nets 11 days ago, again 11 days ago, 11 days ago, so couple of mosquito mats 12 days ago, user has purchased one AC from Snapdeal 15 days ago, T-shirts 18 days ago, some ACs he has already visited 20 days ago, 21 days ago, 21 days ago, and few months like 210 days back he was interested in or maybe he was looking for phones on Snapdeal. So this is the kind of raw data that we have, if we can summarize this data, the user is interested in bottles and mosquito nets, maybe because of summer or rainy reasons these days and he has already purchased T-shirt and AC from Snapdeal and some days back he was looking for mobiles on Snapdeal. So we are going to run plain collaborative filtering that we have already discussed on this data that there is, let's see what are the recommendations coming. So recommendations coming are some ACs, user have already purchased T-shirt ACs, then bottle, mosquito nets, bottles and some irrelevant stuff. So do you think this is the right thing to show to the user in the recommendation section? If he has already purchased AC from us, then there is no point showing him AC because there are very less chances that he'll purchase AC again. T-shirts, yeah, if he has already purchased T-shirt we can show him more T-shirts. Bottle and mosquito net, he was recently interested in is coming little lower in the list. And some irrelevant stuff that might come because of the user behaviors that we have captured through the collaborative filtering but user might not be interested in this stuff. So what are the problems that we have faced with this plain vanilla collaborative filtering approach? One is old purchase that we have already seen that user has purchased AC and T-shirts are dominating in the recommendation that we are generating. So that is dominating the recent interest of the users. So one important point that we'd like to mention here that is going to be based of our solution that we have developed. The one is that the user interest decay with time. If user was interested in iPhone or maybe other phone seven months ago, so there is no point recommending him similar products today, right? Because if we revisit the collaborative filtering technique it simply works on a feedback capturing mechanism, right? So feedback, so like this is well-proved at Netflix or Amazon, like for Netflix they are having explicit feedback for a movie of user. But for us like Bottle, we can't ask every user if he is visiting something that rate this product or rate your interest for this product. So we are having a challenge in capturing the external feedback. So how we are working with collaborative filtering techniques we need to capture implicit feedback and that is static for all users. Like for that what we are doing is so we are having multiple activity types on our website like user visit, user might have added product to wish list or cart or user might have purchased product from us. So for these all activities we give static score like one to view, two to maybe wish list, three to add to cart and four to purchasing. So these are like, so based on that we can generate recommendation but these feedback mechanism is not capturing the time. So which is the key here because user interest decay with time and might be, so user might be interested seven months ago in iPhone five but today he might be interested in iPhone six. Undelated category product that we have already seen are that some chapel and all other products are coming on the recommendations. Thus user might not be interested in all those products. So no point, so if user has like, if we are having user 10, user's 10 activities in a particular subcategory or particular category on our website. So we don't want to show him products related to all the products, all the activities that he has done on our website. We just want to show him some products in that particular category. So this is again a challenge while working with collaborative filtering. Though we can overcome this but let's see how we are doing it. So first going further into this discussion let us see how we are capturing data at Snapdeal. So we are having product beauty line. So that is a unified data platform where all kind of events on the website, mobile apps and all that comes into a single platform. So for us it is a single point of contact to get all the data. Another major thing that we face, the challenge that we face was like, so from the history what we have seen is user tend to log in just end of the workflow while only checking out, they tend to log in. So the activity like for visit or anything that happened in between, we don't have that those activities are captured against cookie IDs only, not emails because we don't have user logged in with our website. So migrating those activities from cookie to email is a challenge in itself because there might be a possibility that same browser is used by two users or single user might be using two browsers or multiple machines or maybe mapping mobile and web data to a single email ID. So for that we are having one intelligent system in place based on the user history or user buying pattern or user visiting pattern. This components map these activities to their email IDs. So this is how we generate data for our recommendation process. So this is how, this is our recommendation solution is, so we are having a lot of data, visit data, wishlist data, car data and purchase data but as I've already discussed we don't want to generate recommendations for each and every activity that user did on our website. So for that we have written one component called seed generator that creates or that picks the best item in each category for us. The best item how it calculates, so we will discuss in the coming slide but for us getting best seeds in every category or subcategory is the, this is done by this component. So then we have two further component, one is exploiter and explorer. So in the exploiter, the name gives you a bit of negative but in exploiter what we try to do is we just try to generate recommendations based on the user activity that we have. Here we just try to expand the seeds that we have. We try to show similar products to those, those seed items, so there is nothing that we are doing anything, machine learning or anything stuff here. This is just we are exploiting what user did on our website. In the explorer part, what we are doing is, let's say if user is inactive for multiple days on our website, so there is no point showing him similar products again because if seven months ago he was interested in iPhone 5, so there is no point today showing him in Google 5 or something like that. So for us, but we do, so in a coming slide what we see how we do this explorer part, but for that what we do is we try to capture user interest and based on that we, on the user behalf, try to explore products. Merger is a component where we merge all the results coming out of exploiter and explorer. So we are not generating product based on the ranking mechanism, we are just taking the global score mechanism so that product takes their own place in their recommendation feed. So that is somehow taken care by the merger or we will see it in a detail in the coming slide and from the merger we do generate the recommendations. We do some redundant removal kind of stuff in the merger and we do generate recommendation out of that. So key terms that we are going to use very often in the coming slides, one is categories and subcategories, so you all must be aware that the entire product catalogs we are having around very big, 12 million product catalog at Snapdeal. So we have divided that catalog into categories, subcategories and more granular level but for simplicity sake, we can just think of divided into two things. One is categories and subcategories under a category. Second, based on the historical data that we have, we divided subcategories into two types. One is where single purchase can happen, like you have seen the example of AC. So from the historical data that we know is people generally try to buy a single AC in a particular season. So based on that knowledge that we have with our data base, so we divided our categories into two parts, one is single purchase and other is multiple purchase. Where multiple purchase can happen like t-shirt, user can buy multiple t-shirts, research time. So based on the historical data that we have, we found is how much time a user generally take to make a decision before buying a product. Like for AC, somehow we find that it's 21 days. So we know that till 21 days, the user is going to research in that particular category, so we want to boost that category a bit. So that is what we're using research time there. Expiry time, so every product has its expiry. So based on our repurchase behavior or the historical data that we have, we are finding a term called expiry time so that we can utilize that to remind the user, okay, you can buy it now again. So the first component that we have is clean seed generator, where we try to pick best item in each subcategory. So the main idea behind picking up the best item is, so again, the concept is user dynamic interest. User interest changes with time, so we use this fact here and and try to find best item based on the exponential decay function, okay, what kind of item user would be recently interested in. So it also use multiple component that I've already discussed, research time, recency is also a key there, recency that I've already discussed and subcategory affinity. So let's say in a particular subcategory, some item is coming out of best seed, right? But in multiple categories, items might have same score or we may want to boost some item. So for that, we are using a term called subcategory affinity with some formula, we calculate user interest in that particular subcategory. And based on that interest, we try to score products, so score seeds accordingly. So again, so we are using exponential decay with time because we know the interest decays. So from that raw data that we've already seen, where user was recently interested in bottles and mosquito nets, he has made two purchases, one is AC and another is T-shirt and user was interested in iPhone or some phones seven months ago. So based on that, this is our seed that we have picked. For that we are going to generate recommendations. So you can see here is the mosquito net, user visited 11 days ago and the bottle user visited nine days ago, but bottle is coming down in the list in comparison to the mosquito net because of the subcategory affinity of the user. So user tend to buy in this particular category a lot. So we want this to be ranked higher, ranked or scored higher than in comparison to the bottle. Purchase AC, so it belongs to the category where only single purchase can happen, but we will see how we are utilizing this space. T-shirts, again, multiple purchases can happen and the phone he has visited. So these are in the order of their score. So another component that we have discussed is exploiter. Here we just try to exploit user input that we have. So for us the user inputs are like the seed items and the seed items that we are, sorry. So here the seed items that are generated out of seed generator and we are expanding. So we are, how we are finding similar products to their seeds, the first component we are using is collaborative filtering. Second, so we are using image, so we are using item with similarity of collaborative filtering to similar items for that item. Another thing is content and image based similar products. So what we want to do is based on their brand, their price, their description and all that stuff, we want to find similar product based on the content. And image based similarity, also we are using in some particular degrees because in every category image based similarity doesn't make much sense. So the third one is cross selling. So we, based on the market basket analysis, what we find is what items goes well together. So based on our data that we find is for frequently bought together products and that we use to cross sell these products based on the purchase that user already made on our website. Candidate recommendations are coming out of this component. So we have one rule engine in place, like we have already discussed. We are having two purchases happened with us. One is AC and that belongs to the category where only single purchase can happen and the T-shirts where multiple purchases can happen. So for the category where only single purchase can happen, we are utilizing this space for cross selling. Like for AC we are very much interested in showing users a stabilizer that goes very well with AC and for the category where multiple purchase can happen, we utilize the space for both similar product and the FBT that goes well with that product. Global weighted scope. So what we want to do is, so we are having around 50 million users with our database. More than that, I'm not updated with the numbers, but the context here is we are getting only 1 to 2% of users that are coming daily on our website from the entire user base that we have. So for every user, so we don't want to generate our feed or recommendations for every person that those are inactive for a long period. So for that we want to run our components, exploit our explorer on a different frequency. And for that, we want to have global weighted score in place so that we don't want to rank each and every product every time, but what we want to do is we just want to give a score and based on that score, the product takes their place automatically in the entire recommendation feed for that user. So this is the exploiter function that we are using. So this is a function against score and activity times, score decays with the activity times. So how old are the activities, score would be less for that. Because we want the recent products to come up in the list. System-generated companies. So we are having another component in space. We want to utilize for, like if we want to do forcefully recommend something based on our business strategy or some other factors. So initially we are using it for predicting user repurchase. If user have already purchased some item and let's say we are notified by the system that the expiry of this item has already reached and we can ask user to rebuy it from us. So again, that idea is right product to right customer at right time. Explorer. So here, so if let's say user is inactive a long time on our website. So the data of that user that we are having with us is kind of stale, right? If user was interested in something seven month ago, or eight month ago, so there is no point showing him similar product again. But we don't want to lose that information of that user we have, of that user we have with us. And we just want to compute here the category affinity of that user. So that based on that affinity, we can recommend him the trending product in those particular categories today. So if user is coming after seven months ago, so if we are having data like he was interested in iPhone five at seven months ago, so today we can recommend him iPhone six for that matter. So if user is, so that also applies if user is not coming in a particular subcategory with us, or particular category with us. If user is not doing any activity in phones category today, so we also want to recommend him other the products in that particular category, but obviously they are having low score because of the recency of other activities. So exploiter versus explorer, since we are having global weight is core mechanism in place, so explorer versus exploiter graph automatically takes place for a, let's say for a longer period, if user not coming into that particular category, so in that particular category, explorer takes a lead than exploiter, so explorer, product recommended by explorer are coming top in comparison to the product recommended by the exploiter. So we simply calculate here user affinity, based on our formula that includes interest decay with time, discovery top trending products in each category based on the sliding window algorithm, based on the product launch date, visit date, number of times it has been visited, purchased, and that that changed very frequently in a day multiple times in a day. So based on that, we try to update the explorer component of our feed multiple times a day. So this is how, again, it's a decay function, decay with user, decay with time, and so again, if the activity is older, then it decay with time, the score decay with time, merger. So since we have generated a lot of recommendation from our different components, so it's time to merge them, and you can see that exploiter is generating some recommendation candidate, explorer is generating some recommendation candidate, though they have their own global score, so we need not to do any ordering or anything here, we just need to club them, and but what we also want to do here in the merger that remove redundant products, like if same product is coming from two seeds, maybe because of FBT or anything else, we just want to find overly similar products over there and remove them, and other thing like sold product, inactive product, we take snapshot of our inventory system on very frequently, and we try to update our feeds from the already sold products in that category, and based on that we generate recommendations. So this is how exploiter versus explorer graph works in a merger, so you can see that till 50 days for that particular instance, exploiter was taking a lead, glue is exploiter, right? But after 50 days, explorer takes a lead, so explorer products that are coming as a recommendation from explorer will be having a more score than exploiter after 50 days for that matter. So based on this algorithm, let's see what's the feed that we are going to propose. So this is our feed. So we know the user was interested a lot in mosquito nets, so we are showing some mosquito nets in the top in the list. User has already purchased AC, and that's two few days back, and still in research period of that particular stabilizer. So we are showing him just after the mosquito mats, the bottles, genes because user has purchased T-shirt from us, so genes goes well with T-shirt, so we are showing him T-shirt, iPhone 6 because trending today against the iPhone 5, that was trending seven months ago. So this is the feed that we are going to generate. What are the technical challenges that we have faced while building these solutions? Again, the big challenge was data because that the algorithm, if you see collaborative filtering content, image similarity and all that, that takes us a lot of time because those are kind of n-square problem. So that takes a lot of time and we want to update our feed frequently in a day. So for us generating, so for us like writing the entire whole solution in a single goes was not a good thing to do. So the components like image-based similarity, content-based similarity, collaborative filtering, that we run daily. So because a lot of products on, because a lot of things are not changing over there. So, but the feed exploiter and explorer pod, we want to run multiple times a day for inactive users because if user has purchased something at 11 a.m. in the morning, we just want to show, or visited some item at 11 a.m. in the morning. So we just want to show him similar products or explorer products, just at maybe 1 p.m. or something like that because we are not real-time yet. So the challenge here was like, so everything, so the core here is the everything decays with time, right? So today if any, so if today on 17 July somebody did an activity, so tomorrow it's going to be a one-day role and day after tomorrow it's going to be a two-day role. So for us calculating decay every day was an issue. So for that what we did is we take a reference rate, let's say 31st July and we are calculating decay against 31st July. So till 31st July, for the July month the decay will remain same. So these kind of challenges that we have faced and again we need not to run the feed for the whole data till 31st July, but at 31st July we need to run a full feed with all the raw data and all that stuff and to recalculate the user decay and all that function. So the incremental behavior gave us a lot of challenges. Optimization and handling data screeners, so we are having again I have told you that we are having 12 million big product catalogs. So for us, but all those products are not sold on a daily basis. So for us while generating recommendation, data screeners was a very big problem. So and we solved this by doing a lot of things like grouping all the stuff that were technical solution that we find. Tweaking algorithms to increase the relevancy. So like we have already seen the decay function that we have. So those are not very much that simple while calculating because we need to some cap like for a particular item if user visiting today we don't want to score get beyond 0.7 or something like that. And the lower cap also we want to restrict to some limit so that we cannot run off infinity or any kind of problems. So for that we need a lot of tweaking. So let's say also we want to do something like that if user has purchased AC yesterday, right? And after three days back he's visiting same product again. So we don't want to consider it a revisit because user might be showing it to some of his friend, colleague or he might not get the delivery yet and so he's just visiting that product again. So here we just wanted to treat it as a purchase not a revisit again. So this kind of solution we did by designing our decay functions accordingly. So that for a particular period the purchase will take its lead. So we are having a lot of component in place. So for us the challenge was orchestrating multiple flows. Some let's say explore runs six to eight times a day but explorator runs four times a day and some other products like system-generated recommendations run once in a day. And so we need to orchestrate all those flows and generate warning message if data is not available or maybe stop the process or kind of stuff. So that we have handled through Uzi. So what is the new stuff that we are going to work on? So first is again the motive behind is personalized touch for users. So for us, like we are having multiple components already written like user classifier and product classifier. So based on certain attributes we have classified users and classified products on the similar attributes. So for what we want to do is we want to map so user and products are based on that attribute. So for example, let's say if we somehow find that this user generally purchase when deal is going on. And we also find some product that goes well in deal only. So while in the deal period or sale period we try to show user those products only. So we are working on that. So we are having social data, we are gathering social data of users. We are trying to utilize social signals of users to show him the recommendation product there. Friends are buying, visiting or doing an activity on that. So we want feedback capturing mechanism in place so that we can come to know how people are liking or responding to our feed so that we can optimize our solution. Again, the thing that we want to do is real-time recommendations. Since we need to current data maybe in batch but we want to build a model on that and try to show user real-time prediction maybe similar products or something like that that we want to do in real-time. So this is it. From my side, do you have any questions you can ask? Hello, Gagandeep. You mentioned there is a research, yeah. You mentioned there is a research time window considered as 21 days for ACs. Like considering that the user would research for this period of time. Now, given that the product catalog that you may have would be in millions, how do you identify the time window that you may actually want to narrow down for a given product item or a product category for that matter to decide research time window because this cannot be automated as a process. So how do you identify this? So for us, there are multiple factors. We are having category stream that guide us because they have market study or thing in place. So we don't have data for all 12 million products. So we are having categories teams that provide us inputs and we do have a component in place that helps us in validating those recommendations that they make. So do you mean it's manually driven based on human insights? No, so it's first they have given us but yeah, we have system in place to validate whether. So those are like based on the market research that they are having, right? So what we are having the system in place is where we validate whether this recommendation is right or not and we update accordingly. Okay, thanks. Hi, this is Vijay. Do you collect the information in the server side for every request or through JavaScript? Can you come again? Do you collect all the information in the server itself or through JavaScript up to the page loads? The reason I am asking this is, will you be able to figure out all the robots access and remove them from all the data collected? So if I understand your question correctly, so how are we serving these recommendations? No, what do you do with the junk data? Maybe like visits by robots, et cetera. Okay, okay. So again, the system that I've discussed beauty line. So all the scrappers or the junk data, the scrappers, cookies, and that we are getting. So we are having system in place to remove that from the rod. Okay, but that is your own build solution, right? Yeah. Okay, and are you also capturing events on page load like for a single visit? He's moving mouse over for a few things. He's trying to collect. Yeah, so for that, so we are like, for that we are having some components, but for us, getting data daily from those components is not feasible today, but we are working on getting dual time and all that stuff. But that is not in place today. But you're collecting and using in some cases. Sorry? You're collecting the user click behavior and mouse behavior and using in some cases. Okay, thanks. Hi, I have a question. So you mentioned that you have users from different channel and you use mapping from cookies to email ID to identify the users from different channel. But as you said, you have about 50 million users and I believe that number of actions that user does on the site would be very less. So by using the user action on the website, I feel that accuracy would be very low because 50 million users and number of action are like, would be very less compared to that. We are not mapping it to number of actions. So you said that you see the action, sequence of action that user does on the site. Yeah. And based on that you identify user uniquely when you're mapping cookie with email. No, actually the thing is, let's say for a particular period if user is not logged in and we are getting some events from that user or particular cookie. So for us is to map those cookies, events that are coming through cookie to their email ID, right? So if we are not getting email ID, so we are having a lot of like component, a lot of things to get this email ID. Like, so for us in a log we create a time series, right? And we find when the user was last logged in and when the last log out happened. And so for the event those are not mapped to any user in between that. So based on the learning of the user behavior pattern we try to map those cookies to the multiple user that are coming, those are coming through those browsers. Yes. So even if I log out, will you be able to identify that it's me? So if user, let me give you an example. So if user A logged in, right? And did some activities and user A logged out, right? Some activities happened from that cookie. We don't know whether user B did that or user A did that, right? It's fine. Which is fine. And after that user B comes, user did login from that particular cookie and logs out after that. So the events between the user B logged in and user A logged out are not mapped to any user right now, till. Yeah, so that means. So based on the behavior of the user searching pattern or user visiting pattern or user buying pattern that we have we try to map those items to the users. Same user. So based on that we find whether it belongs to user A or user B. So you're not able to figure out, let me come again. So user A is logged in, okay? He does some activities then logs out, okay? And then maybe he or someone else comes to the same computer does some browsing. So these browsing after log out, will you attribute it to this user who had logged in? So we try to divide those events between user A and user B because user A might have done those activities after logging out, right? Okay, okay. My question is towards a trending catalog. Hello. Yeah. So you mentioned about collaborative filtering. So my question is about trending catalog. So how do you show trending products and make sure that kind of a bubble doesn't happen? For example, you show a particular, there are like few hits for a particular product and then you show it as a trending item. And again, the hit keeps on increasing because it is shown as a trending item. So how does the trending catalog on those pieces work? So you are saying some product is a, continue to be a drop trending that particular category? No, basically I'm asking about the algorithm or the way in which you show products as trending and make sure that the bubble doesn't happen. For example, after reaching a particular point, maybe that product was not liked by people but it was viewed so many times but that should not be shown as a trending item, right? So this is very a non-problem Britney Spears problem if you see. So we have handled it, a lot of papers are there you can read. So we are, how we are handling is we try to identify these items. So in that particular category, so we try to see how other items are behaving, right? If it's being a top for let's say a couple of days or couple of weeks, then we try to solve it accordingly. Because for us, like other items in that particular categories are again gaining the interest of the user. So for us, those are more important than the product which is a trending for last week or last month. Okay. Okay. Yeah, I'm referring to the bias. Sorry? So basically, I mean... Maybe if you are not done, we can discuss offline. Okay, yeah. Okay, Gangandeep, thanks for the insights. I have a question that is not actually related to the recommendation systems. But like off late, we came to know that Mintra has shifted its entire business to app. So I'm not the right person to answer that question. Okay. Any insights like from Snapdeal? Like... No, nothing. Thank you. Okay. Business Insider and all that own open platforms that they are to understand. Hi, Gangandeep. My name is Sunil. I'm a product manager. And thank you for the informative session. Thank you. I just want to know how do you measure the performance, continuously measure the performance of your recommendation engine? So performance in terms? So whether it's working itself. So for that, we are having AB system in place. So we are running multiple algorithms accordingly. So at a time, we run two algorithms. And based on that, if you are rolling out new changes or anything for that algorithm. So we try to find how users are responding to that algorithm. And again, I've already mentioned that we are having in our wish list to having feedback mechanism in place. If user is coming and clicking the second product on our recommendation feed. So for us, it's an opportunity to analyze why user did not click the first item. Or maybe, so I can tell you, based on the historical data that we have, so what we have seen is a user try to click the second item most in the list. So we are using these kind of user behavior. Utilizing these kind of user behavior in our rule engine and all that stuff that we can configure automatically. Yeah, every company has, yeah. Hi, this is Amol. Actually, my question is something like there is a purchase which has actually happened from your recommendation. The first ranked item is bought. And it's a single purchase. So how soon you will going to be removed from the existing recommendation result? So right now, I'm not, so we are not real time. But yeah, we do run this component four times a day or five times a day. So based on that, let's say if user did purchase at 11 AM, so one PM that the first run after that going to happen is going to remove that product. Okay, so currently what is the turn around time? And replacing it with the similar products, yeah. Currently, what is the turn around time? Turn around time. Like if I bought the first ranked item from the recommendation, how soon it will disappear from the existing recommendation? So what I'm saying is we are running it three times, we are running five times a day. So you can divide it in 24 hours and see. 24 hours of, thank you. Sorry. No, no, no. We just run for active users and update. So we are having incremental system in place. We do run only for active users. We have time for one more question. For the implicit feedback, like visiting and adding it to the basket or purchase, you said like you mark it as one and two and three. Is that arbitrary, discrete numbers? So this is an example that we have used, but based on how the feed, how these algorithms, these functions are behaving, we tweak this because these are working as a wait for us, right? For us, if you want to make the decay more and so based on that, we are tweaking. So you have a confidence score for each of different event. Basically, purchase is treated as more weightage than the visit. So that's what we want to do, because if you think from the perspective of the user, for us, we are capturing feedback, right? So for us, if user has purchased something, so user was more interested in that in respect to the product that he has visited. So the way we are capturing them now. So this is static, nothing related to confidence level. Okay, thank you, Gagan. Thank you. And with that, we come to the end of the 2015 edition of Fifth Elephant. We hope you love the conference and you come back again. Thank you.