Hey folks, I think we're gonna get started My adapter somehow My adapter somehow didn't make it with me and so I I have an older machine So I'm gonna start without slides and if we can the magic volunteers can figure out the Right cable in the right place, then we'll switch to slides So I'm here obviously to talk about floss the future and the internet so a good place to start is where we are today and Floss is everywhere today, and and that's a real success Which is worth celebrating doesn't really matter where you look you find free software at the heart of the internet at the heart of the big services that we use and in all the projects various applications so Seems kind of obvious today, and now free software and open-source software are just a kind of part of mainstream development but you know that's really a surprising thing and When I sort of started in this field You know that wasn't the case at all and there was a real disbelief in free software and open-source software as Be incredible in any serious sense I think there's some learnings from that that will be useful in the current state and so I'm thinking back to just around the turn of the Millennium So 1999 2000 when Mozilla was newish about a year old and and I spent a lot of time Explaining really basic questions like what is free software? Why would anyone participate in free software? Why would anyone do anything where they didn't get paid? I mean these always seem to me really silly questions given how much time people spend doing things they love People would also ask Well, how can you do anything without employees and titles? You know how how could it possibly work? Why would people collaborate? How could you share something and why would we ever trust it and? and When you take all of those together, you know, it's clear that that people who hadn't been involved in Floss and in that era, you know, they couldn't imagine it They were living in a different world and they couldn't conceive of something different And the world they were living in is actually not so dissimilar from the one of today it was pre-internet of course, but if if you think today of Facebook and Google and The AWS side of Amazon and you roll all that up into one company That was the world we lived in in 2000. You know that company was Microsoft and it was deeply deeply hated but it was the way things worked and so the Even you call them the tech nerati or the press or the thought leaders or most software developers Really had trouble imagining how the world could be different And then the free software and open-source movement began to grow and to begin to move Up the stack of developer tools into an operating system Into more complicated parts of the stack and eventually even up to the consumer product level Now that was a surprise, you know all the time we were developing Browsers we were told open-source and free software could never be good enough for consumers That I was always too rough around the edges And it was really only for people who were going to spend their life using that but but in fact the free software movement reached a new level and Open-source and free software started to become mainstream and Then then the conversations I had changed and a whole new set of conversations appeared And they were questions about taking the ideas of floss and applying them to different parts of life And so there would be questions about I want to do open data or open science or I'm an open architecture project trying to crowdsource designs of habitable homes that you know almost no cost for people who are homeless or You know open medical records or open health care, you know you name it the the wave of fields that suddenly realized shared development Collaboration not owning things making knowledge available became immense and Also the idea of collaboration or sharing or shared value began to become mainstream You know in that era the idea of sharing cars whether its Owners sharing their cars through uber or zip cars or shared cars was simply unimaginable The idea that you would have a bicycle in a city that a city would own a bike and allow its citizens to use it crazy I mean didn't didn't exist. There was zero of it in the in that era and so open and free Software began to influence whole ranges of fields And so those fields aren't you know software per se and don't have exactly the same criteria But the effect of a small set of technologists thinking differently and acting differently and building something different became profound but at the same time something else was happening and And That is free software and open-source software were at the center of Internet products But increasingly other aspects were being added Around them and originally it was other software All you know according to license so not not unexpected But but because the software is not distributed we have free software and then where there's a fair amount of modifications made to those which are not returned to the commons and then layers of Algorithms or or activities which are not captured or included in in floss software So there's a whole layer of algorithmic Thinking behind that and then of course mass amounts of data that don't live in a floss like system and So when we look at the internet today, you can see that there's floss at the center and Many more layers of closed or proprietary Unknowable activity surrounding that and so although floss has been wildly successful at being at the center of things We live in a state where sort of closed and proprietary systems prevail And at the same time the human experience has been changing, you know in the early days Everything is very positive and optimistic and we thought of the web as we called it then as being You know the enabler of collaboration and empowerment and freedom and knowledge sharing and all knowledge for everyone in the world And a range of very positive things Which is part of it? But increasingly, you know, we see other activities and the human experience is Sometimes positive and sometimes increasingly negative and Today, you know one question. I'm asked really frequently as well. The the internet is broken, you know, what what can we do? And that's a complicated and tough problem that has you know software elements and data It has probably some regulation in it. It needs a lot of innovation. There's a whole set of systematic changes to Build a system that's as powerful as it is and also healthy but there's one element to Making better life online that I think rarely gets attention and that I hope We can make clear and more obvious and more active and that is floss communities and floss software because The set of what I think of as floss DNA It is a powerful tool for building solutions to the problems we have today And when I say floss DNA, I mean Three or four things one. I mean free as in freedom Which obviously is a you know comes out of the problem of the English language that that we don't have the word libre And so we need to distinguish free as in freedom, you know from free of charge and that is You know if floss software is free as in freedom the internet today is free as in free of charge and the idea of What the freedom to actually empower yourself and act looks like is not in the dialogue today And so the four freedoms of free software may not apply directly to how does an individual live But the concept that you can get engaged That you can participate that you can see what's happening that the rules that apply to me apply to you Those core things that drive the freedom of free software are not part of the internet experience And many people can't imagine what that would look like and and when you talk about Individuals may be being able to participate or see their data or use their data the responses often Oh my gosh, that's so hard. That couldn't happen. What would look look like nobody would care Nobody would do it. Who would bother? How would you design a system and those sounds sort of daunting? but in fact, it's very very similar to the kinds of questions people used to ask about free software and so this question of How do we make the idea of participation and the ability to act? Right, so with software consuming it's not enough How can I how can I actually touch it and change it and act is a core? Question we face at a broader scale both at a product development and development level online and for You know the consumer and citizen as well So that's that's one area of floss where the background and experience in living Could be effective in helping solve today's problems And also I would say open as an open source not open as it has come to mean today Which is the powerful can grab whatever they want and do whatever they want with it That seems to be what I hear open what People think it means but open source, you know, I mean lives by the same licenses and definitions and and lives with a Set of equality and ability to participate The second area You know free software creates shared value That's partly why we create it is because it's valuable and it's valuable for for people of Different viewpoints and perspectives and goals. It's not single centralized This is good for me and everyone else lives in this system and today like the idea of shared value or interoperability or Competition or real sort of link ability back and forth or if people being able to take parts of a project and use it Is not part of online life We were back into a closed system and so once again, you know The idea that of having lived these things and and built the kinds of software that the floss movement has built is a Powerful tool in solving these things you know floss communities also have a value of collaboration rather than the command and control of You know one organization owning and deciding everything so another clearly needed value and The last one I would say is earning leadership Which is closely related to integrity and trust, you know in a very different model from being an employee, you know and being told what to do and the floss world has a Just such a broad range of how to earn leadership and what it means and how you keep leadership And how you inspire people and how you make something effective and so You know it is pretty broad range so for example at Mozilla, we do not have you know leaders for life some projects do you know as a as a Mark of the founders and respect and so on So we have a broad diversity. We don't at Mozilla. I feel like My job as a leader is to continue to earn that leadership and at some point if I'm not Doing the best job or I don't have anything unique to offer then it's time for me to move on And and if I don't do that voluntarily then you know, I think the project should have the right to Make it clear to me that some you know someone else would be doing a better job I mean, I'm not looking forward to that but but I think that those sorts of practices and policies are not common in current society and that the experiences that we as a broad Community bring have immense value And so where would we aim those things? Well one key problem today, you know is online you are the product and so There's a current understanding that you need immense profiles to deliver ads and That the conventional wisdom is oh well. We couldn't I mean ads run the system and you have to have these vast amounts of data in order to sell ads and I'm really Interested in exploring exactly what information you need for what purposes so for example We are running and you know, we have a product at Mozilla. It's called pocket And and we're experimenting so I want to start. I'll say it many times. We have no profiles No profiles no profiles no profiles pocket is about Identifying reading that you want to take some time to read like what's content that's worth attention and reflection and so It's a simple tool, you know, you save something to read it for later turns out like that doesn't happen that often online a lot of times people see a headline and and and Share it but but actually aren't our reading material and so what we're doing with pocket Is to use those signals? Oh Yeah, oh That's okay That's okay, I can do it. I'm just wondering Probably shouldn't go backward, huh? This is it so on the pocket Left and right There we go. Okay, so you're going to see some of the older slides, right? This is the set of questions This is the open floss at the center You know the layers that I spoke about being added around it I'm just trying to get to the next set of slides. These are the This is a little slow here closed systems Experience on the internet used to be all sunny now. It's you know, we have these very dark Concerned like what's actually happening? What do we do? Want to answer Like this is something we are not in the discussion at all Like who are the sets of people and who has ideas and lived and built something different than the conventional software before Floss values free is in freedom. This is the You know a single entity ringing every last dollar out of the system no matter what the cost system we're living in today with the Sort of alternative that the floss system lives with which is shared value for many the collaboration piece I talked about earning leadership and Okay, so clearly there's a set of key problems on life. What might we be able to address? So item one, you know, you are the product And so this is the the combination of ads Which exists everywhere? So, you know both personally and a Mozilla. We are not anti ad I mean they're annoying and I certainly like life without them But most of radio and TV and magazines and newspapers, you know ads are everywhere and they support a ton of systems So we're not ready to say somehow magically. We're going to make something new that doesn't have ads I mean, I'd love it, but I don't I don't know what to offer But what we are adamantly opposed to is the current tracking system that goes with ads and this question of profiles So we have this experiment with pocket, which is can we deliver interesting long-form content to you Customized to you without profiles because the conventional wisdom is you need a profile for anything and the more data that you get anywhere The better it is and so far for long-form content at least we're finding that yes We can actually deliver Personalized content to you without a profile now we do that by having the logic on device And so information is delivered to your browser or your pocket wherever it is on your phone And and that that device obviously has a lot of information about you and it makes a selection But and you as a user can say oh, I don't ever want to see anything like that again Make it all go away or you can say oh, that's interesting I ignore it or you can you know choose to read it Or engage in some fashion and so what we're finding is that the information that's on your device Which we never see and don't collect Don't use to track you because we don't have it is Quite Acceptable for delivering content that you might enjoy so that's an interesting experiment We're not like that's not to say you can deliver ads. I don't know You know that's a different set of experiments That we would like to see lots of people being able to engage in and then there's a kind of profile Which is useful For encouraging people to do all sorts of things And and that's the kind of profile that sucks up all the data about you And uses every last thing, you know and uses that profile knows probably more about what motivates you Than you do so we've seen for example Use an example out of the United States probably Russian of Targeting very specific groups of people with the goal of getting you know X percentage of this group to come to a demonstration because they're outraged about acts and At the same time targeting another group of people to come to the same demonstration Because they're outraged about the opposite And so the actual creation through micro targeting based on profiles of a civil protest and So that's a special kind of profile, right? That's the and the conventional wisdom is that gathering everything up This is like what this happened on Facebook So just to say that like yes, you need all that information in order to be able to do socially useful things But we don't actually know that like is it socially useful to outrage groups of people? So they go out on the street and fight each other over something that isn't actually real Like probably not so I think it's a very interesting question of What do you actually need to run? I'll call it mainstream commerce You know whether it's content or even ads versus how much information is being collected about us now and used to manipulate behavior and I think if we could build more products that Tested more of these things it would tell us whether some form of information gathering is just socially terrible right and and that the business of targeting at a level of manipulation where you can intentionally create violent disturbances might be treated differently than a profile that was adequate for e-commerce and That requires a lot of testing right it requires research on an academic level and it also requires You know products that are on the like how much Information do I actually need to do something legitimate and so I mentioned product pocket because it's one Sometimes if I say that without an example people think oh you're kind of crazy or what would that mean? but but we have one in the market that's You know actually well received and so that I think is a power of law software Is to build and test those things and demonstrate that a different kind of product and experience is possible Originally these had transitions So that's why there's so many of these slides in here but a second problem is clearly content and misinformation and how do we even know what might be reliable or Whether it fits our criteria of trust or not And this is another area where experimentation in Software and products is important. So we have a little experiment here, too called the Facebook ad analyzer Which is currently forwarded by Facebook. So We'll see what happens, but but what this you know, there's Facebook has information about Here's an ad Here's the organization that's paying for the ad and here are the targeting criteria and so this is an election cycle in the EU and We thought it would be helpful when you're Seeing ads about EU candidates to be able to You know select an ad and say well a who's paying for it And why am I seeing it? Am I seeing it because I'm a woman in a certain age group living here or am I seeing it? Because I went to some particular site about immigration and got tagged as you know pro-immigration anti-immigration pro-ex so This is because that transparency of political ads in in the EU is highly regulated and is seen as an important piece We've been trying to bring some added transparency to the current election cycle But Facebook broke that in the last couple of days. And so this is clearly an area that that requires either the Big platforms to be interested in it or some kind of regulatory Engagement now fortunately the EU has been involved in this and there is a code of practice And we believe that Facebook actually agreed to make this information available And so we've asked the EU Commission to look into it and see if our understanding that this ad Information should be available for the for the political ads And so this is an area of product experimentation Which is harder to do unless you're involved in policy or unless there is some policy or practice in place But it's also an area where the EU is you know world leader And so we think that there's increasing possibility here And I know like in the early days of floss like for the first decade of Mozilla. I Did not want us engaging in things that required policy activities that because they're hard. It's a different skill set You know, but about a decade ago. I thought the time had changed So so Mozilla is active in policy and the good news is there are a lot of I would say internet activists who are real estate policy Driven so in these questions of well, I want to build software. You know, I don't want to deal with codes of practice or Regulators the good news is there are many people out there who do and what what they really need is connections to technologists and so The will and energy to do this is very high particularly in Europe and developing better connections between I would say not just technologists but the floss community in particular and these advocacy groups would help us a fair amount and The third big area Really to aim at I call the addiction economy Maybe I'll go back on that one Because not with the you know, the designers of internet experiences what they're aiming for is to make people happy Because when you're happy you stay on their site. You engage in their site. You do the things that generate revenue for them But it turns out I think research is showing that what really The like that fight-or-flight response the way you activate human beings quickest and fastest is with fear and outrage so even with the motivation of people being happy to stay on their site a Lot of the most effective ways of doing that are appealing to the I would say More difficult sides of human nature fear anger outrage violence. And so the addiction economy given human nature is Very closely tied into I lost a few of them But it's closely tied into these practices that we're seeing where the degree of Violence and anger and disruptive behavior online is is really quite high Okay, and so that that leads us to the question You know, we saw this a little bit in open source communities early on where when people came online You know, you sometimes you'd find really aggressive and nasty behavior and and women in Floss communities early on were targeted quite aggressively and the overall level of discourse was a little nasty At least for people who came into it, you know at Mozilla This was a constant problem as we were trying to introduce new people into The open-source world and they would be a wow, you know, what you're trying to do is so nice But the online discussion is so harsh Well, it turns out that has zero to do with free software or open-source communities That's for some reason what much of humanity does when we first encounter these tools Like the harshness comes out and so you see it and like how common death threats are online today You know, and I would bet real money that If and when we track down, you know these deaths and you know rape and torture threats Many people will be otherwise Upstanding citizens that somehow feel free online, you know to engage in this behavior so figuring out like what is it that encourages something better than that and How do we break this cycle and Here again, I think free software and open-source communities have a lot to offer because we've been dealing with it for a while and the quality of discourse in our communities has changed and the standards of behaviors have changed and community practice and Community etiquette and what is acceptable and how do you deal with the troll or a poisoned person? Like we have a set of experience in those things that is certainly Imperfect and we have far more to go But nevertheless, it's a stage ahead of where most people trying to deal with these kinds of questions are and so Just as we were early in the sharing and collaborative and freedom aspect the how do you make a community? What's healthy? How do you get to better behavior? We have some experience with it and I'll go back and say I'm Quite convinced that technologists have a huge role to play in how do we create Software and products and online experiences that are Encouraging better aspects of human nature Currently the economy is driving towards the addiction economy where you're the product and where we're gathering so much information that you're easy to manipulate and So I think Absolutely requires Technologists to be involved and to be committed to that and so in the last few years I personally and for Mozilla felt really compelled to make that commitment clear compelled because You know the open system Open Internet the World Wide Web You know the graphical layer which is open open source free software like these are so fundamental and so important and yet not enough They are not enough to create an online life. That is decent. They're not enough to stop trolls They're not enough on their own To be an alternative to systems that encourage sort of outrage and violence And so I felt quite compelled that that at least I and testing, you know Mozilla Felt equally committed to something different and so we've made some commitments, you know Mozilla is clearly it's an open-source project I think the speaker before noted were imperfect But Committed to open source We have a manifesto That talks about security and privacy and individual empowerment and just felt we needed something more And so we set out four additional things about the internet that are important to us, you know one that it's inclusive It's the wrong button here You know internet experiences software and products currently are Encourage outrage. What would it look like if we were actually trying to encourage civil discourse, you know and rationality and thoughtfulness? You might not maybe be the richest company on the face of the earth But that's not what really floss aims to be and then finally We are actually committed to trying to take that value of collaboration and shared good and trying to extend it Further from the floss software itself out into other areas And I was happy to find that you know, not just me personally, but that this sort of resonated within Mozilla So we've adopted it as part of our core sort of manifesto and how we think of of who we are And it may or may not be quite right for you Because floss is really varied and we have lots of diversity among us but I am hopeful that as a Community of communities this question of can we actually build a better online life Is is increasingly important and that we actually can represent ourselves well in the overall discussion Which is lacking in technologists When you at least in the in the worlds that I travel in or it is hard to find a large Loud technical voice other than the large platforms So Mozilla is very welcome when we show up and that's great And there's a lot of diversity and floss so it would be great to see others as well and then finally the last thing that Mozilla does that not all Floss projects do is really focus on the consumer Because you know, that's most of humanity and that's where there's a certain kind of impact Like that's a messy place to be as a floss project Because you have to take what you're trying to accomplish as floss and in our case with the manifesto And yet build a product, you know that consumers want Which is generally free of charge and convenient and fun and interesting so it's messy You know and and causes us to be you know imperfect in many ways But it is a piece of the puzzle that we try to bring to the overall sort of floss world And so to circle back to the core point That floss communities floss DNA and the floss principles Can be like a powerful force for improving the state of internet life And I hope you find that to be true and I certainly hope you find if you're interested ways to be effective at that And if you were trying to do such things, you know I certainly offer myself and Mozilla to try and pot provide whatever I don't know support or assistance or cheerleading or Anything else that we can do to help more people be successful. Thank you And I think they're not here, but I want to say a special thanks to the volunteers and crew here who made up for my lost adapter so Just a small announcement. There are still five minutes for questions. So if you have a question, please Let me know I will Give you the microphone for that and please if you are leaving be quiet Still the session is is not yet finished. I I have a question on okay On the on dominance practically you mentioned that Large corporations are dominating the internet. So on a technology side when you look at the browsers We now we see today that practically The domination is Chromium in Floss sense or Google Chrome and It's it's not Mozilla anymore. So So I'm and I'm guessing that most of the people here in this room also use Chrome And I'm a little worried about that. So what what do you think about this kind of domination on the desktop? I would say Yeah, so the question about the risk of domination in the browser space or the desktop Yeah, I think it's quite real I I do want to admit that there were some years there in which Chrome was a better browser. I Think that that time has passed and certainly technically You know, it's great to have the crown again And so it is a question of why use Chromium Or why use Chrome? You know, sometimes Google breaks things except for Chrome It's only a day or two and it's always some low-level product manager that just can't be controlled So maybe that's why but I actually am interested it'd be interesting, you know to know I think it's a real risk And I what I really think is a risk is what we call the web Like so, why do we care so much? Yeah, I love Firefox, right? But but really why do we care so much? We care because Right now Google is almost free to do whatever it wants and call it the web and it does Like it makes up API's sometimes. They're useful sometimes. They're really Google specific sometimes They have tons of problems. There isn't like there are standards bodies, but you know with its market share It doesn't really matter and so the reason that we care about Firefox Yes, there's a product piece and we love it But the core reason is the web and we are about to lose the web like inner operable Multivendor Platform, I mean we don't think of the web now But the technology that we call the internet the graphical layer all the stuff that you can see and that gets delivered through a browser Engine whatever app you're using soon. That is about to be Google Again, the only thing standing in its way right now is Firefox or gecko and it is not at all clear that we are gonna succeed I should be loud and clear about that the question of whether we have what we used to think of as the web is The question right now and we are investing a ton and it is so expensive because for some reason Google has Accelerated the number of people in development and stuff. They're adding in in a way. That's very hard to keep up with And and and I guess it's good for developers, but that question like the next year may tell that question Right and so we're still committed because you know, we love the web I mean, but but something that's interoperable like we're just on the edge So I don't anyway, I guess thank you for asking because I clearly care about it so a Couple of years ago at pos damn 2014 I believe I went to a great talk by a guy called pool Henning camp who gave a satirical talk about the state of online security He speculated that the influence of the NSA and gchq was deliberately weakening online security And in the face of such powerful influence against our sort of interests And the technical solutions were ineffective He concluded a talk by saying that this is a political problem. This is not a technical problem And I guess you could make a similar argument in the sense that such heavily heavy Influences Google and Facebook and kind of face those interference in elections You could say that well in the face of such heavily heavy influence boss technical solutions are ineffective, you know I'm sorry. I'm really having trouble hearing. Sorry What would you say to the the the proposal that in the face of such heavy influence by Facebook and Google against Sort of freedom and online privacy that technical solutions are ineffective that this is a political problem well Sort of monopolies are part political part technical problem for one Well, that's interesting is it a political or a technical problem I like well the the issue is it's actually both right now How do you regulate? Right and so that that is part of why I think that the Floss communities are important because you need experimentation to know how to regulate So for example, you know if the EU Commission appeared in front of us and said just tell us what to do like what's the answer Like you could you could say the basic ad model is unacceptable Right. What happens then? Right and let's let's say, you know, like do you want to not have Facebook? I don't know. So so so that question and Then the other political question is sometimes you can't innovate because the monopolies are so powerful I'll love that ad problem that we're having and then that's a political question But it needs the technical voice in there to explain it And thus my pitch for you know If you find the local internet advocates who are trying to advocate for privacy, but don't know anything about technology like You know, even if your inclination is to kind of like leave them, you know A little bit of help on technical chops will go a long way. So thank you Yeah, let's thank our speaker with some


Good evening everyone, welcome to the last session for today on research in databases. We have actually changed the session a little bit because we will be having guest who will be joining us in a few minutes. Professor Sunita Sarawagi who is one of the leading international experts in the area of data mining. She started off her PhD in the area of databases, but after her PhD she moved into the area of data mining and she has been the PC chair of various leading conferences and very well renowned in this area. She will be joining us in about 10 minutes time. So, the goal of this evening session was to have a discussion on research in general. So, last time I had presented a few points on how to go about doing research and I had taken some questions on these. Sure many of you would have more questions and the other thing was last time we did not cover any specifics in detail and some people had been asking me about data mining research. So, this time around I requested Professor Sunita Sarawagi to join us and so you can ask any questions you have on data mining which she will be happy to answer. She will join us, she is here and will join us in a couple of minutes. So, let us start the discussion with questions from various centers. There are quite a few centers that have raised their hands here. So, let us start with that directly. Emity University Haryana, if you have a question please go ahead. Sir, sorry to ask this question, but my question is on normalization sir. Sir, I want to make that kind of relation so that that relationship violates all the normal forms. Is there is any real life example for that? For an example that violates all the normal forms. Well if it violates first normal form it automatically violates everything else because that is a requirement for all the normal forms. So, that is kind of trivial, but if you want to explicitly show violation of the other normal forms, just throw in a few dependencies. You know, throw in a dependency that violates BCN. Just add a few attributes and then throw in whatever couple of functional dependencies and multi-valued dependencies as you please to show violation of the other normal forms. That is your goal. It is not too hard. It is a small exercise. Just take five attributes and then add a few functional and multi-valued dependencies. So, you can use this Taylor to violate one of the normal forms and you are in business. Does that make you happy? Is there any real life example we can give? Real life example. Yeah, you could translate these back to real life, but you know, what is the point? If you come up with a really bad design involving real life attributes, sure.  So, you can take these attributes, do it in abstract and map it to some real life meaningful attributes. It is not all that hard. It is a good exercise. It is just a fun exercise. There is no real thing involved here, but maybe something you can play around with and show your students. Any other questions? Thank you, sir. We have Maulana's NIT, Bhopal. If you have a question, please go ahead. Sir, what is categorical database and how can we mine the categorical database? As there are so many algos, but not any, everybody is perfect. Not a single is perfect. Is there any way to solve this problem? Okay, so you are leading with a data mining question. I will invite Prasasunita Sarawagi who is here to join us and answer that question. So, let me introduce Prasasunita. So, Prasasunita Sarawagi, as I mentioned just a few minutes back, is one of the leading experts in the world on data mining. She has worked on many sub-areas including graphical models, OLAP and data mining intersection and doing data mining through SQL among many other things. And she has been the program chair for the leading data mining conferences in the world which means she recognizes one of the leading experts. So, let me transfer the question to her. So, actually I will ask you back a question. So, what do you mean by mining categorical data? What kind of mining were you trying to do? Data mining. As there are so many algos, just K means K mode. Oh, so you are trying to do clustering of categorical data. So, actually for clustering categorical data, I mean clustering in general, you know it is sort of not perfect and it is very much tied to the application. So, what is the goal of your mining exercise and what are you hoping to get out of it? What is your objective for this mining? You want to mine categorical data means with data which is ordinal, nominal. Yeah, so you could so there are. So, for the if the goal of mining is to discover something new and serendipitous which you had not expected, then you can apply clustering or association rule mining. And for clustering, so there are many algorithms. So, if you look at say, so there is a long time back there was a paper by Prabhakar Raghavan on clustering categorical data and then there are many forward and so you know in general I will give a meta answer. So, if you know of one good paper on clustering categorical data, you should sort of stick in that papers title in Google scholar and look for forward citations to that paper. So, look at papers which have cited this paper and then you can find about other algorithms which are possibly improvements over the algorithm which you are trying and which is perhaps not functioning for you. So, I am sure there are lots of such algorithms actually professor Sudarshan will ok maybe the network is not working. You can continue I will bring this up in the background. So, you will find other algorithms and you know if you have categorical data and you are trying to look for some frequency values, then sometimes you are better off just doing explorations manually by loading the data into an OLAP tool. Professor Sudarshan must have talked about OLAP. No, I did not. So, in his book there is mention of OLAP. So, in the advanced topics you can read up about OLAP. Sometimes you might be able to sort of find something under greater control if you use such OLAP tools rather than depend on some black box algorithm. Hello ma'am, I am working in textual dataset and I have some problem to find how I am convert unstructured data into structured format. So, you know there are a whole range of techniques available for converting unstructured data to structured data. Sometimes you might just be able to write some very simple rules to do the extraction. And if your rules are slightly more complicated, you can use software like gate to help you write those rules. But if your data is really, really unstructured and rules will not work very well, then you can use some statistical methods. And there are many software which now you can download and use for training such statistical methods. So, you can use. So, Andrew McCallum has a tool called mallet. And then. So, a whiteboard if you want. So, statistical extractors. So, I am just giving some options. So, one option is to just use Stanford's NLP package. Write a bit slowly. So, another is to use mallet. So, a third option if you have a lot of the infrastructure around extraction available to you and if you just want the core engine, then I have also developed a software for extraction. But it does not have a lot of the supporting routines which for example, Stanford's NLP package might have. So, that is the CRF toolkit which you can get from CRF.sourceforge.net. And so, for statistical extractors there must be others that I cannot quite recollect now. There is also an open NLP software package which is quite good. Open NLP, I do not know if there are two n's or one n in that open NLP. So, you can just search for this term in Google and you will be able to find it. So, for the statistical extractors, I mean you should use them only if your rule-based extractor is not able to handle the variety of data that you see in your source. But unfortunately, even here it is not like there is one golden software package that you will use and if that does not work then nothing else will work. It is not that. It is a matter of trying out different things and something will partly solve your problem. Ma'am, these tools are free available in websites or this purchase. Sir, these are all freely available. License tools. And before these, you can try gate. So, this is so something. So, these are the statistical extractors. But before that, if you want to just try with some start with something which is very classical, which has been around for a long time. So, this is the rule-based total environment called gate. So, you just type gate NLP tool and you should be able to find the pointer to it from a UK university. Ma'am, the capacity to convert unstructured data into a structured format. So, how much capacity it will convert into a structured data? So, if you consider typical rule-based extractor, it will be quite fast. But the statistical extractors will be slower and it depends. If you are looking at numbers, what can I say? I do not have exact numbers on top of my head, but they are not very fast. I mean, most of them will be slow. They will be much slower than taking an HTML page, converting it into a DOM tree and indexing that page. Suppose, if that takes time x, then these will take anywhere between 10 to 100 x time. Recall precision. So, in terms of performance, you mean accuracy. So, if you are just talking about accuracy of extraction, then if you want to say, think about standard entities, like say identifying people name and organization names from things like news articles, which are very nicely formatted English text. So, from there you can get pretty decent accuracy like going to 90 percent also. But if you are talking about extracting even the same kind of entities from not necessarily news text, but arbitrarily, you know, arbitrary plain text documents available on the web accuracy might be much lower, might be like even 70 percent. For any other kind of new entities that you create, you know, it will of course, you know, it is very subjective. It depends on the type of the entity and how much training data you give for. Thank you, ma'am. We can show. Yeah. So, actually I was talking about using Google Scholar for finding out about more algorithms on a particular topic. So, I was mentioning this one paper on clustering categorical data by Prabhakar Raghavan. And the reference was not exactly sort of specific in my mind. And then Professor Sudarshan did this search using Google Scholar and he found this specific paper. And now you look at for that paper, there is a link called Cited by 246. So, if you click on this Cited by link, you will see the papers which have cited this paper. And now you can find, you know, you can find out, you know, any other article which also has high citations and which looks like an improvement over this algorithm and try them out. So, forward citations are extremely useful in helping you sort of figure out if there are sort of better methods than what you have been trying out. But of course, these days you might find lots of papers on a topic. So, in general a good very rough good criteria is to look for papers which are more recent because then they would have discussed all the other previous work hopefully. And also the ones which have a decent number of citations themselves and appear in good venues like top tier conferences and journals. We have Mahatma Gandhi Mission Noida, please go ahead. How we can create data cube in Oracle? I have not played around with data cube facilities in Oracle specifically. The SQL standard has a cube by clause just as you say group by clause, you can add cube by. That will give you relational output which has all the groupings. For those of you who are not familiar with this, let me use the white board and explain it. Supposing we wanted to analyze sales of items by certain properties. So, let us say that we have a relation which has every single sale that happened. Let us say sale ID and then we had various attributes which are called dimension attributes like may be city, then the let us say date, time and may be amount. Now, date and time or may be this is time of date. Let us just keep it simple and you know analyze may want results such as select city, date, sum of amount that is the total sales of whatever items were sold in each city on each date from sales group by city, date. So, that gives one particular aggregate, but there are many more aggregates that are possible. May be we want to group by city alone, may be date alone, may be city date and time and may be a number of other factors as well, may be based on what was the customer who bought it. It was this frequent customer, occasional customer, may be some income level of customer. So, there are many more things in here and you may want to group by various subsets of this. So, the SQL standard has this cube by construct city, date, time. So, instead of group by we say cube by and essentially what it does is it creates a union of a large number of group by queries one for every subset of attributes here. Of course, if I have a group by city and up here I have date what is the value of date you know there is no unique date here. So, that becomes null. So, that is the cube by operator in SQL and I am pretty sure Oracle supports it. Beyond this Oracle does have OLAP tools I am not familiar with those I have not used it. May be Sunita has something more to say. No, I have not used I have also not used Oracle's OLAP tool. I have used SQL servers long time back like say 12 years back and that time it was quite useful and it also had an Excel plugin. Yeah, I am sure things are much better now. There is a you are discussing about just aggregation means you are yes sir, sir you here you are discussing about transaction level data and over that we are using here aggregate function that is the OLAP and data cube. But I my question is what how we can implement data cube by using Oracle. Like a start a schema is not like a schema and fact constellation we can implement it in Oracle by using this primary key and foreign key constraint over the fact table and dimension table. So, that actually implementation how we can implement. I think your question is how do you do schema design for a warehouse on top of which you can do this cube. Is that the question? Yeah. So, we have been looking at various methods for doing schema design, ER modeling, normalization and so on. Now, for data cubes the way you design the schema is quite different. I do not have a whole lot of practical experience in this. But the general idea is that most of the applications where you need these kinds of things the schema kind of presents itself in the sense of having some sales relation or some other such thing one or a few fact tables which are you know there are a large number of records here. This is the core around which everything revolves in this in the case of OLAP. And around this core you need to build various dimension tables. So, in my quick and dirty example I had date and time. But it is not just by date that you want to do grouping you might want to worry about quarters. You might want to worry about financial years. You may want to dig deeper and have a monthly analysis. So, you can build hierarchies on this. So, those dimension tables and their attributes are driven by the business needs. Now, there are some things which are obvious once you have date most of the OLAP is for business analysis. So, the month quarter you know financial year all those things are very obvious candidates. Similarly, on day of the week is another obvious thing if you want to worry about weekend demand versus weekday demand. Obviously, for a business that is actually selling products the demand varies widely between these categories. So, there are various hierarchies you can build on this and then. So, those are all part of the dimension tables and then you can define cubes not only on the underlying date for example, but also on the hierarchy in the date week and so forth.  Typically, it is fairly simple OLTP data in large volume and then a whole bunch of other dimension tables which work around it and the dimension tables themselves you should not usually apply normalization. So, for example, if I have a dimension table date that is has one entry per every date. Now, given the date based on the month and you know the day of the month and or other such things I could always use a lookup table to find out the day of the week the quarter the financial year and so forth. But I am also given let us say quarter maybe I can well this is not a good example, but in general if you have a city ID for example, from that the city ID would automatically determine the state there is a functional dependency from city to state and from state to country maybe states have unique names across countries. So, if you did a normalization of the dimension tables you would split it up, but in the OLAP context people specifically avoid normalization for efficiency because the goal here is not to you know you are not updating OLAP data directly the updates are happening somewhere else and you are just copying it into this and redundancy is a good thing here in the sense of if it can speed up your query processing redundancy is perfectly fine there is nothing wrong with redundancy. So, for the dimension tables there is no attempt at all made to avoid such redundancy and the focus is on efficiency. So, the design of a OLAP schema is quite different from the design of a traditional relational schema. I hope that is partly answered your question if you have a follow up please go ahead. Sir, one more question I want to ask can I ask? Yes. My next question related with the graph mining when we are using graph mining then. So, how we will store graph in database by using Oracle? So, there are a lot of tools for storing very large graph which have come about in recent years which fall in the big data and even and the goal is to partition the graphs across machine. So, this is the research work which I am familiar with. Now, just storing a graph in a database at the core is very simple the schema for it is straightforward. The issue is how does this gel with data mining? So, my experience if you store you know graph in the database and then you want to do querying on it. Yes, you can issue SQL queries to go and fetch data, but it is not necessarily efficient. So, for other graph related stuff for example, in the work on keyword search on graphs which we have done if you had to go do SQL queries each time to access you know node and edges out of the node you are dead it is too slow. So, the only realistic thing was to extract the. So, you store the graph in the database for persistence, but when your service starts up it loads the graph from the database into memory and has an in memory graph representation and then works of it for keyword query. I am pretty sure the same thing would hold for any graph mining algorithm where the graph is small enough to fit in one machine's memory and also to be analyzed efficiently using few CPUs which are connected to that memory. But today graphs social network graphs and so forth have gone way beyond this. So, you have to partition the graph somehow. So, there has been some work on it including some work which we did a while ago in a slightly different context which was keyword search on graphs which are larger than memory. Our goal was not to partition it across machines although that was a side effect, but how to you know keep the thing in the database in some special form and then load parts of it on demand, loading just the minimal parts needed to answer keyword queries again. So, we worked on that that is where I have some familiarity with this area, but there has been a lot of work subsequently on graph clustering and search algorithms on graphs. So, one of the relevant things here let me write this down. So, there is a system called pregel from Google which is based on a very old paradigm called bulk synchronous processing of BSP. So, the basic idea is very old, but pregel is a recent implementation of this paradigm which lets you do a number of queries on graphs and where the graph is split across many machines. So, if you look up the pregel paper you can find more information on that. Just search for pregel and Google perhaps that should get you to that paper. Again as before scholar.google.com is your friend gives you a lot of information about paper. So, actually there is for graph mining specifically a graph lab is another software of a platform which is supposed to do a better job with irregular graphs. So, graph lab it is a software developed in a university. So, I do not know whether it is as good as a platform that you will get with pregel, but it is definitely worth exploring graph lab it is from CME. Anyway you cannot get pregel, pregel is not open source, but there are some clones of pregel which people are working on. So, I think there is some Apache project which is cloning pregel I do not remember the name, but it should be out soon if not already. But yeah that provides a platform it does not implement any graph mining algorithm on its own. So, all that has to be done on top of this platform. I do not know how it provides any support also for graph mining. No, I do not know about now, but graph lab my student has been experimenting and it is sort of he found it better than I do not know with pregel what soft whether he tried and he just did a theoretical comparison. So, graph lab is also worth while trying. They definitely implement belief propagation is basically an inference algorithm on a graphical model built on top of a large graph. So, I have not used Mahout specifically, but there is this Apache Mahout project. So, Apache Mahout is a set of programs if you will which are targeted at mining of big data. So, this sits on top of Hadoop we will be covering Hadoop tomorrow for those of you are not familiar with it. Hadoop is not designed specifically for graph mining it is a parallel processing paradigm based on this thing called map reduce. It is a parallel it is a paradigm is map reduce Hadoop is an implementation of that paradigm and Apache Mahout is set of tools for data mining in general which sit on top of this. So, they can work on very large data. I suspect there would be some parts of Mahout which target some kinds of graph mining. I am not familiar enough with it to say that for sure, but I suspect it would be that. My question is related to data mining especially like association rule mining. Like as we know when we apply association rule mining we do get a lot of redundancy. So, can you give an idea like how can we apply a zenitical law of thumb for removing this redundancy especially the fitness function which will be the beta fitness function to remove the association redundant rules in the association rule mining that is being generated by any algorithm for association rule mining. So, the answer I do not know much about genetic algorithms, but there are many ideas which have been proposed since the original association rule mining paper on removing redundant rules. So, one set of papers again you will have to use the same trick as I was telling about this categorical clustering on searching through Google Scholar, but there is one set of techniques which have been developed by Haike Manila and I do not remember the other author. So, which talks about how to use essentially the very well defined statistical notion of the measure of surprise of a particular item set given the observed frequencies of subsets of the item set to filter out high dimensional patterns which are not surprising given the lower dimensional patterns. So, that is one set of techniques. So, in 1998 I and with another colleague we did we also did another sort of dimension of work along a different dimension on getting rid of redundant rules and that is by exploiting time. So, any rule which holds steady for a long time is assumed to be known by the analyst is therefore and is therefore not interesting. So, you want to modify association rule mining algorithms assuming that with each transaction you also have a time stamp available to output those rules which have interesting, but steady variation along time. So, that is another piece of work. So, in general though if you start from that Haike Manila paper and then there is another paper by Rajiv Motwani and Geoffrey Alman which is sort of which started this line of work on finding interesting rules and if you look at forward pointers to that paper you will find lots of other work on filtering away the many redundant, but not so interesting that association rule mining produces. So, there is actually a very huge literature on the topic that you have asked. So, you will have to do quite some reading, but I cannot comment about the use of genetic algorithms. I am not familiar with genetic algorithms and I am not also a big fan of genetic algorithms. We can take questions from some other questioner. MPS TME, Shirpur. Within the same application can I use a special query as well as a normal relation query. How can I use it? In fact, that is not at all uncommon. So, for example, I may have shops which sell certain things and so I have relational data about what shop sells what things. So, I made filter out shops based on some relational predicates and at the same time I want spatial information also I want things which are close by. So, as long as your spatial data is in the relational database and the database supports these queries. Yes, you can certainly ask queries which have both regular relational predicates and spatial predicates. So, the oracle spatial extensions certainly support these combinations as far as I know. I think Postgres also has this post GIS as far as I know it is not very efficient, but if you just want to try out these queries it is a perfectly good platform. So, I think that also supports spatial predicates integrated with SQL. So, it is pretty straightforward to ask these questions which combines spatial and regular relational predicates. So, the next question is can I use in the same application more type of databases for an example I am using oracle db2 and postgres in same application is it applicable or not is it possible or not. Ok, it is possible, but not highly recommended there may be some historical reasons for it. For example, in IIT Bombay we have financial records on oracle database because that is what the TCS built and our own academic records which we built the system for are in postgres. So, the moment you do this there is always an issue of keeping the two in sync. If you add a student you better add it in both databases, you add a faculty member you better add it in both databases. If you give a scholarship which is recorded in this student the academic database it had also better be recorded in the oracle database. So, all of these have to be there is more work, but if it is needed for other reasons it is perfectly feasible. You just have to decide what all data is going to reside in both and set up mechanisms to ensure that the two are kept in sync. Sir, one last question is there. Suppose I have already created database in SQL which is of extension dot bck and now I want to use that database in postgres where the extension is dot backup. So, how to do that? You want a backup created on one database to be restored on another database. I do not think that will work. You can do an SQL dump that is where the system outputs a bunch of SQL you know create table insert table command and so forth. Those are little more portable, but otherwise I think there are commercial or what may be an open source. I have not used them, but there are tools to export data from one database, convert the format and then load it into another. But I am not familiar enough if you do a Google search you will find these tools. Thank you sir. Thank you very much. We have Shree Jayachamar Rajendra College Mysore. Please go ahead. We use Hadoop and MapReduce. Can we use it for Aadhar sir? Your comments. Can you use Hadoop and MapReduce for Aadhar? So it depends on what kinds of things you want to do. So if your goal is to use Aadhar to validate a person who has come to a ration shop then Hadoop and MapReduce play absolutely no role. But if your goal is to do some analysis on the Aadhar data maybe to even check for duplicate somebody who has registered twice and so forth then maybe these tools would be useful. So it depends on what you want to do with it. These tools are good for distance support and certain other kinds of bulk querying but not for you know small transactions which do just a little bit of work. Those are not the right tools. There are other tools for that if you want to have a massively parallel database. Aadhar is large and I am sure it has to be split amongst multiple databases. Does it qualify as really big data from the viewpoint of validation? Maybe not. It is fairly easy to break it up into multiple data you know to partition it horizontally. The hard part of Aadhar is to see if people are applying twice. That is easily the hardest part. Maybe Sunita has something to say about it. Yeah that is actually really a hard problem and even in general given a database of size n and if you want to look in that database all possible duplicates that might exist. You know theoretically you will have to do a quadratic time computation but of course you cannot afford that. So you know but it is a very expensive computation. You might have to take maybe at least you might have to compare a record with 10,000 other records. Actually maybe in the case of the whole Aadhar database it might be more like 1 million. Yeah so it is a very expensive operation. So you might have to there are many algorithms which are available for doing duplicate detection in large databases. Some which involve which assume a pre-existing index. Others which sort of create multiple signatures and they only will sort of compare record groups within records within a group which have the same signature and then there are other sort of you know blocking techniques where you create signatures where the order makes sense and you sort your data based on that sort of you know order sensitive signature and you do a sliding window kind of search. So there are many algorithms which are sort of like a fuzzy joint kind of algorithms which you will have to implement. Yeah and in such cases map reduce yeah you know if you use the signature based methods you could use map reduce. We have eBit group in Tamil Nadu. Is there exists any widely accepted algorithmic approach for performing outlier analysis on large datasets. I will defer that to Professor Sunita. So actually outlier analysis is like clustering. It is basically the complement of clustering. So there is no one single algorithm which will work for all kinds of data. There are many algorithms around I mean there are I mean for example if you have if you are looking for outliers in OLAP kind of data you would use a very different algorithm then if you are looking for outliers in say you know just categorical transactional data like see the ones which are kind of biased by association rule mining. So yeah I would not say there is one widely accepted algorithms. In fact you know it may be even less so than for the case of clustering where maybe k means is that one widely accepted algorithm. Now in general for outlier detection the the meta rule which is followed is that you create a model. That model can be a clustering model and then an outlier is anything which does not very well in that model. So with that rule you know with different instantiations of model and the models will be chosen based on the kind of data that you have you will have different algorithms for outlier analysis. So I have myself not done too much work on outlier analysis for general data but I did work on outlier analysis for multidimensional OLAP kind of data and there we actually built a sort of a multidimensional model like you know ANOVA kind of model with time series modeling and we looked for outlier in that OLAP data and there that model was a very nice fit and we found interesting outliers but I would not use that model for some other kind of data. It was specifically chosen for OLAP data assuming that the number of dimensions is not very very large. So you know if you have some specific kind of data in mind maybe I can elaborate more but in general no I do not know if anyone will accept it. One more question pertaining to queries, SQL queries. It is assumed that queries, SQL queries involving Cartesian products is always imposing a lot of overhead on the amplification side. It is better to it is suggested that it is better to write queries or replace queries which usually does not involve Cartesian products of tables. This statement is how far this is true sir let me let us want to know this. I mean is it recommended to perform or write queries which involves Cartesian products or there should be queries which are free from Cartesian products of tables. So I cannot think of any pretty much any meaningful domain where you actually want a full Cartesian. There are a few rare cases where you for example if your goal was to do for loop over several things and you want to code that in SQL then there are some cases where there can be Cartesian products. So we cannot rule out queries with Cartesian products saying that they are totally useless. So for example if I have a year table with you know with years each row is one year 2000, 2001 and so on and another table with months with the things being January, February, March and so forth then a Cartesian product of this would give me a month, year combination which I might use to do something else later in the query. So occasionally Cartesian product like this could be useful but it is rare. Most of the time you know Cartesian product unless you intentionally did it is probably an error. You forgot to put a joint condition. I don't know if that answers your question. Okay thank you. GRIAT Kukatpalli, Andhra please go ahead. What exactly the difference between federated databases and data warehouse? Okay so federated databases is a term which is generally been used to mean that let's say you had a number of databases within your organization and sometimes this happens because two organizations merge. Air India and Indian Airlines merged which was a big mess of course but the fact is that they would each have had their own computer system and if you want to merge the computer systems then one of them has to dump its system and move all the data to the other and that might cause some problems operationally. So maybe they continue operating their own computer systems just like they continue operating Airbus planes and Boeing planes and so forth. But now the issue is that you do need data from both of these databases. They need to see each other's data. So federated databases where each of them continues to operate as a separate database but you provide some support such that you can have queries run from one of them. It's allowed to access data from the other maybe even do lines with data from the other and so forth. So that's federated database and then you might even allow transactions on which span these things. So the databases are still separate but you provide some kind of view on top which lets you do queries and updates across the two databases. Does the functionality save in both? The two continue to run exactly as they did for their own functionality. What you do is you add some more stuff on top which lets them interoperate with each other. So if you have a query new query on one which needs to access data on the other it can do that. If you want update transaction which runs on one but also needs to update some data on the other then you can do that. So that's what a federated databases they still have their independence but you allow certain operations to span two or more databases. Last question sir. Can you suggest any sites for I mean same question just I asked in the previous also regarding spatial data mining. Can you suggest any sites for getting the data sets on spatial data mining? Yeah so I think I said I will provide a link to sites I have not had time to do that. Let me just put this down so you can search yourself. So there is I mentioned a few things there is US data. These are things you can find this easily. There is a tiger data set which many people have used. This is for spatial query processing but I am sure there is a mining aspect. But the other thing is you can get in touch with Professor NL Sarda who heads the GISC. I think that is geographical information systems engineering or something lab here at senior colleague here at IIT Bombay. So if you search for him you can also I think they also have a web page for this lab. So they have been collecting data sets from various places. I do not know if they can share those things but you can certainly contact him and get more information about availability of data sets from Indian sources because he has been working on that. Other than that I know that there are many many more data sets of road networks and other such stuff which are publicly available. So you can search for them. I think European road networks there are some data sets. There are many others. I cannot name specific ones but it is not too hard to find some kinds of data sets whether that matches what you want to do. If you want to look at a particular data mining problem you know does a road network data set help you at all maybe not. I have not worked in this area so I cannot directly point you to things but I can help you search for these later. Thanks a lot. We have Rajalakshmi Engineering College Chennai. Please go ahead. Hello. Good evening sir like this question regarding association rule mining. So while applying association rule mining we have been getting a lot of overlapping rules and redundant rules. Is there is any specific tool or algorithm to do that? Maybe so this question was asked just a while back and I gave answer to that question. So let me write down this time so maybe it will you can then search on Google scholar. So there is basically the first paper on this topic was written by you know there were other authors but I remember two of the authors. Rajiv Motwani and Jeff Anman. So there was a paper which do not remember the title but definitely there was association rule mining in it. So this was maybe written say roughly in 1997 I think. And I have to write a little bit of it. Oh ok yeah ok. So this was one pointer and then there is another sort of paper by Aki Manila which also talks about interesting I am not giving the exact titles but just some keywords with which you can search association rules. Yeah but tools so if you search and I will come to that later so this was I think in 1999 and then I have done some work on this with some colleagues when we were in IBM so this was the paper with Chakra Perthi and Dom. So this was a paper on temporal association rule mining so which talks about etc. And now you talk about tools so you know this is there was a question I saw which was sort of put in that which was just asked on the online. So in general if you just go to www. kdnuggets.com you can find pointers to many data mining software tools and I think most of them which do association rule mining would also now have some support for managing the very very large number of rules that a typical association rule mining algorithm would produce and since these are now sort of you know many years have passed since some of these papers were written I expect that this will also appear in the tools which you can sort of search on this website. Can we association rules for this exam or mining XML documents for what documents like XML documents like so far we have XML documents. So association rule mining has you know the input association rule mining is a set of sets so it does not matter whether your document is in XML format or in any text format you have to convert your data into this set of sets format. And then you also have to think whether it makes sense to run association rule mining on the set of sets that you have created. So you know it does not matter you know what in what format you know in what format the data is stored really it is basically you know you think about the abstract input that is required by association rule mining operation and see whether you can you know whether your XML data actually contains sets of records where each set in turn can be thought of or it can be converted through some simple transformation into a set. Narayana engineering college Nellur please go ahead. Sir I am having one doubt in related to the human behavior analysis sir so can you suggest me best algorithm which is suitable to analyze a human specific behavior. I do not know anything about that area Sunita would you know. Human specific behavior you know if you look at the www.conference www.conference they have you know if you are talking about human modeling user behavior for browsing you know it depends on what kind of user modeling that you are trying to do but if you want to consider user modeling for says on web browsing so that you can find out you know what a user is trying to do within a session you know there are lots of you know over the years you will find you know there is some or the other track will be talking about human modeling I mean but I am not familiar with this area I just remember seeing such track titles in the www.conference you know this conference so if you sort of look for you know you know every year you will find you can just look at the set of papers which have been published in that year and essentially search for human and then look at the papers which talk about some or the other aspect of human modeling. Thank you madam thank you very much. We have Bansal Institute please go ahead Bansal. My question is what verification methods of research findings in technical research except mathematical proof are acceptable as research efforts. This is a deep question what is research effort what is not so that is a very hard question to answer in general to me research would be anything where you have some general finding depending on the venue where you submit your research the expectations of what is novel vary so I do not know if you can have generic definition across all areas and so forth but in computer science you know in the olden days research was primarily about coming up with new algorithms for doing stuff but I think there are also other kinds of things which may be acceptable for example if you there are a lot of algorithms but for a particular domain may be finding out which algorithm makes is the best fit for a domain might be considered acceptable research in many venues as an example there are lot of tools for teaching now if you are doing a study of which tools work well for teaching students you know that may not count as research in a computer science conference but in an education conference that would probably count as research so what is research partly depends on where you are targeting it so there are certainly conferences which look at these things and it is important when you decide to use a particular tool you might do it by gut feeling on the other hand your gut feeling may be wrong we put in all these fancy tools in the place and in the end the students learning may not be helped at all so was it worth it this is the kind of things which people do study and it is a valid topic for research I don't know if that answered your question you feel free to ask follow up questions sir I have few more things to ask in fact my prime concern was that you see I am talking about the technical research so in that area suppose we can take an example if we are talking about some database design or some database design so if we propose some model or design so what the initial proof we have to put forward to get it acceptable in the research community we can take an example of data warehousing suppose we say that the involvement of the user is very important for the successful data warehousing development the background work we have to show it is not always the mathematical proof of anything because you see when we talk of the industry environment where they are working on the practical things and in the academics we are doing the things theoretically so how to bridge that gap that is my prime concern I would be grateful if you can give some idea on that sir so in some of these domains you say what is the importance of getting user input on data warehousing so if you are just doing a study with one particular implementation it is very difficult to say anything much you know it is hard to draw conclusions but in the software engineering domain these kinds of things are common people propose particular methodology and you can propose many different methodologies for doing things but how do you compare them so one way would be to study projects and see how successful they were and try to get feedback from the people involved in the project to see what they felt were contributing factors and so this kind of human feedback might help in establishing that you know doing taking user input in a particular way is very important for the success of that project so in software engineering I have seen papers like it is not an area I am very closely familiar with but I know that there is work which looks at such issues and that is considered valid research in that area it is not mathematical modeling it is taking certain ways of doing things and studying how successful they were when used in a variety of projects in general anything like this you want to establish that it has value beyond one project so it is kind of required to do the study across many things so that you are talking of stuff which has general applicability in a wide variety of scenarios and that would be considered publication worthy if you did something which is very specific to one project and it is not clear how to use it in any other project that may not necessarily contest research I do not know if that answers your question better one last question regarding the significance of white paper which are available on the net how authentic they should be considered and because I have seen these kind of stuff having lots of important things lots of technical coverage in that so are they considered as an authentic reference in case of research that is a good question so white papers are typically put up by companies when they have a product and they want to help you understand what the product is about and sometimes they also do a comparison with other alternatives so these are papers written by companies they can help you understand what is going on but if company exclaims that their methodology or their tool is better than tool of somebody else in their white paper you of course have to question who wrote it this is unlikely I have not seen too many white papers which do these kinds of things but supposing you want to cite a white paper and product X is better than product Y that may be questionable but if you are referring to the white paper to say that this is how this particular tool does something then that is perfectly fine so they are useful source of information they can be cited that is not a problem at all but they are not the same as a research paper which is written by an independent party which may be compares two things but of course most research papers are not written by independent parties most research papers propose some technique and then they proceed to compare their own technique with others so every one of these has to be taken with a certain pinch of salt obviously their goal is to show what they are doing is better obviously they will look at scenarios where what they have done is where their technique beats the others does it mean that technique will beat the others in all scenarios may not be so any paper at all has to be looked at with a certain pinch of salt and that is an important part of research actually if you just accept everything at face value you know then you are not questioning things enough but of course your questioning should be in a way that leads to something new and interesting which you can publish perhaps that would be your goal as a researcher I do not know that helps again thank you very much sir Shanmuga college Tamil Nadu what are the database administration tools are available in the research side and which is most used in the research side database administration tools so by this I the from the research context the tools which which have been developed by researcher are things which automate the administration the goal is to not have humans who are doing the administration let the system configure itself to the largest extent possible so many of the tools which you see today which we talked about you know the tools for index tuning materialized view tuning and other related things they all came out of research researchers said like you know we do not want to have highly paid administrators doing this we want the system to do all this more or less by itself with minimal human intervention so how do you do this then there was a lot of interesting research which was required to make those tools work properly and efficiently now other tools which researchers use in order to do the research I am not sure that you know there is anything which researchers use in that sense the tools which come package with the databases were developed by researchers initially and maybe by other developers later does that answer your question or is there some other aspect to what you asked about Thank you We have Sarvajani college surat please go ahead Sir is there any research effort made in the direction of in memory indexing or hasing in context of intermediate results generated during joint queries on joint table in context of query optimization In the context of what are the last thing you said query optimization Okay I don't know there is a research angle to this the just the in memory indices used for intermediate results have primarily been you know hash indices as far as long as you are saying that the joint are simple equi joints however if you look at more complex joint conditions then it's different in fact it's interesting you ask this question because right now I have an M.Tech student and a Ph.D. student looking at exactly this kind of problem how do you build indices where the joint predicate is of a particular form the tuples have patterns that is they could have an exact value or they could have a value which says all possible values are acceptable star now when you do joint processing when you have a number of joint attributes and number of relations and then these tuples in individual relations can have a mixture one tuple may say A B another may say A star another may say star C so how do you do joints in this context so that's something which we are currently working on there may be other work in this area we have not looked at it in depth yet but yes if you are looking at more than simple equi joints there are surely interesting issues here the other aspect is just in memory indexing not during joint processing but in the context of in memory databases what indexing technique would work best for data which is already in memory you don't have to load it from this B trees were designed for this data there's been a lot of work on this for many many years and there is still some ongoing work some of these have been varying because the properties of memory has changed way memory lines are loaded caching there have been a lot of hardware changes along the way and also main memory databases now become not only feasible but they are becoming common in the real world SAP has released a database called HANA Microsoft I believe is also releasing a version of SQL server which is tailored for main memory many others there are like I believe some 20 or 40 different main memory database tools implementation whatever available there is an interesting talk which one of my friends Rajesh Manjramani has he showed me a copy of his talk which he gave somewhere so there is a lot of work on indexing in main memory so it's an active area still it's a little hard to do brand new stuff because there has been so much stuff to do okay the last two hand okay finally we have the very last question from what is this allury institute varangal yes sir which one is the most suitable dataset in stream mining sir that's a hard question I don't know what datasets are available in the context of streaming I mean there are a lot of applications of streaming data you know the sensor data which are available I know that you know this in Berkeley you know Mike Franklin and all you know if you just look in their group they have been working on sensor data and also I don't know now whether they are like at least five years back there used to be all the sensor data which were available which you could use for streaming applications but again I think the google scholar trick works so I know that Mike Franklin so if I have to look for streaming data what I would do is so if you just search for his name and go to look at his research papers many of them are on stream various kinds of query processing models and others having to do with stream data so now you look for forward pointers to those papers and look at some of the recent papers on stream data mining and see what data sets they have used in their papers very often researchers use data sets which are publicly available and those might be also then you know very useful for you is stock market is it suitable for stream mining madam? stock market data it's the particular kind of data I think it's a very it has some specific properties which may not hold for other stream mining applications but you can certainly use stock marketing data it will be interesting to also explore some non-stock marketing data because stock marketing is a very very stylized kind of application sir one more question sir is there any other open source tool in data mining apart from veka that question will go to madam not sir apart from veka so there is a statistical tool which is very popular in universities called system R system R is a database tool this is also R so this is actually a command line tool so this is it is a I mean I have not used it myself so it's not totally on the top of my head but it's a R statistical analysis package and now what professor Sudarshan was mentioning earlier there is Mahut which is available which is an apache data mining tool it is also supposed to be scalable and freely available so these are the options but you know in general this is kdnuggets.com it's a very useful website to visit and if you click on their software tab you will find pointers to lots of other freely available software like any research issues in object oriented databases object relational data research issues in object oriented or object relational databases were also hot a long time ago again I am not sure if there are any brand new research angles which have come up recently like any area which has been examined in great detail a while ago you need some new angle to it to come up with new research in that area the object relational mapping systems on the other hand have taken off lot recently and they did not receive all that much attention in the research community object relational mapping system so there might be some research angles in this and in fact one of the things we are doing is a project which attempts to optimize database access from applications so the idea is traditional database optimization would optimize SQL queries but if applications do bad things and issue millions of queries there is nothing the database can do it can run each query fast but if there are millions of queries things are going to be slow on the other hand you can rewrite applications to change those million queries into far fewer number of queries which will then run much much faster so how do you automate the rewriting of applications is an area that we have been looking at so we took any Java program and so we can rewrite it to optimize so we have changed the JDBC calls and then rewrite it to optimize database access so there is a very interesting project my last PhD student who graduated some years ago did a fantastic PhD on that and I have another student working on it and several master students that area worked out very well one of the related things is that people are also using object relational mapping tools like Hibernate so we also did some work on how to take a Hibernate program and optimize its access to databases so that work is we did a little bit of work there is probably more to be done so there are certainly issues when these have become real systems people are more worried about issues in here how to ensure that Hibernate's concurrency control mechanism doesn't cause a mess somewhere there are a lot of practical problems which people have with these systems efficiency is one of the major factors so this could lead in turn to research problems in this sub area thank you sir thank you very much okay thank you very much I think we will call it a day thank you for staying back so long beyond the official end which was 6 o'clock so we will see you tomorrow then