Thank you. For those of you who joined us for the science exhibit, this is the second phase of our program for today, and it's a little more fun and something new and unusual for us.  The scientist tells me, but it's something unusual that we are doing for the first time, that we focus on, not necessarily on rice alone, but we're focusing also on life that exists in the rice fields. And it's something that we actually observe quite often and throughout the growing season that you've got plots of birds with us, and it's really amazing the wildlife, the flora and fauna that does exist in the fields. And in fact part of the scientific discoveries that are emerging are that we really need to pay attention to the ecology of the rice field, and that ecology enables us much more sustainable production. Because of course we're worried not only about productivity for today, we're worried about productivity for generations to come. So today we've got an exhibit of photography, and this came about quite in a serendipitous way. One day I was invited to an exhibit of bird photographs by Undersecretary Fred Serrano of the Department of Agriculture. His exhibit took place at campus. And I thought, here we've got this resource, and we do have a lot of birds in the fields. And why don't we invite Undersecretary Serrano and a number of other bird watching and photography enthusiasts on campus and in theory to work with us on an exhibit to show off their work. And so I also knew a number of other exhibitors and you'll meet them today. And we'd like to acknowledge their presence and their contributions just very quickly. First, the Contributor is Undersecretary of Agriculture, Sir Fred Serrano. Unfortunately Fred is on mission right now. He's traveling, but you'll see his photographs and his work will speak for itself. May I also introduce Dr. Kirsof Pares. Dr. Pares is a recently retired professor of economics from the University of the Philippines. And after his long career of teaching young people as turned to bird photography, he'll have something to say about this in a little bit also. I'd also like to introduce one of our colleagues part of the Erie family, Mr. Paul Warden. Paul Warden teaches mathematics and engineer at Brent School, but his passion are birds. And you'll see it as he explains his work. I'd also like for all of you to meet Richard Smedley. Richard is a graduate student working on a dissertation about bird ecology. And he'll tell us about how birds behave in the rice fields. And that's something that we're all learning about as we speak. Also I should not fail to introduce Michael Joyce. Has he gone? Oh, Michael created the bird sounds that you hear as part of the exhibit. And he also arrived at Erie quite serendipitously as a partner to one of our scientists and he came and joined us for this. And finally I should acknowledge our creative team that put together this exhibit. The team is led by Paul Ilario. Paul Ilario is also the curator of this rice world museum. And we shouldn't forget the entire communications team and events and visitors team of Erie are first led by Gene Hettle, who just can't stop doing communications. And Ria Dimapilis, is she here? Ria, please? Can everybody see you? But she'll join us in a bit. She helped engineer this whole affair, put together a lot of logistics that went into the support. Ria, please? So, may I again introduce our Director General, Robert Sieger? Thank you very much, Bruce. You said everything that I thought I was going to say. Well, I'm sure I'll think of something. No, I've been very fortunate in having one of those fantastic office views in the world. My view sits on the corner of the building opposite the fountain. It looks out on Mount Bahau and the surrounding areas. And in the foreground is just a wonderful array of bird life in the rice paddies. And it wasn't always like that. In the 1960s, 1970s, and the 1980s, we, or not we, but those who grew rice thought that anything in the rice field that wasn't rice was a pest. And so there was a lot of spraying and pesticides. And basically the rice paddies in many places were almost deserts in terms of biodiversity. We came to understand the ecology of the rice paddies. And now, in our production fields and research fields, we spray pesticides at a rate that is almost zero. 97%, 98% dropped in pesticide usage in our fields. And what we've seen is a tremendous flourishing of life in the paddies. The birds are at the top of the food chain. And if you have birds out in your rice paddies, egrets and other feeders, you know you have a healthy pad. You have another, you have a healthy environment that insects and frogs and other invertebrates and invertebrates can live quite happily out there and can provide enough food for the birds as well as produce a healthy rice crop. And so this exhibit, the birds of Rerea, the birds of the rice paddy, is in fact the celebration of healthy rice paddies. And it is an indication that the rice ecosystems in Asia can be very healthy systems that will contribute to maintaining the very important biodiversity. So this celebration is really a celebration of biodiversity, a celebration of what we've learned over the decades and what a healthy food system is all about. So I want to congratulate our colleagues here who have put this together. You see them out at all hours of the morning, five o'clock in the morning creeping around, looking for birds. They're a very strange lot and we don't want to go into that very deeply. But I love them and I admire them and wish I had the patience and the dedication to be as persistent in tracking our fine feathered friends. And again, thank you so much for putting this together. It is a stellar exhibit and I'm sure you'll enjoy it. And we do have cocktails and snacks for you. So with that I'll turn it back over to Bruce. A presentation on facts and myths about birds by Richard Spencer. Well, thank you very much. I think we'll start by thanking to be asked to talk. Thank you for being part of this. And thank you to the tune direction we've already had. I'm pretty much saying I think I've got to bring it down on my cards. So what I'll do is I'll start by giving an introduction by myself. I'm, as you said, I'm a PhD student here, based from the University of Redmond in the UK. I'm here looking at the birds and it's absolutely amazing. But first and foremost I link myself as a bird. The birds is what draw me here and this is what really draws my research every day. As we mentioned, getting up at the ridiculous time in the morning to go and see what's around is one of the most exciting things you can do. And if you can ever get it as a job, I highly recommend it. But unlike other people here, I must say that my background is as a bird and so I do very little on rights. So whilst I was still in the UK, I did a lot of research on birds in rice fields and actually for a geographical area that provides quite a lot of the global market rights, there's very little research which can be found. Most of which comes from the US or Europe. So with this, this gives us a huge plan of canvas especially within the Philippines here where area is mostly based and with the partner countries as well. Where we can look at kind of what we can do with rice fields both benefit birds and the farmers. So with no actual kind of literature really as it were or no literature on the birds in rice fields, I went to the farmers in the field and asked what their knowledge on birds was. And this opened up to the largest myth which has already been mentioned once already is that all birds eat rice, which if you take away nothing from me, from this evening, it's a complete myth. That's completely the untrue. They seem to have been convicted, without any evidence. People see birds flying away and they think they're eating the rice when that's just not true. What you're about to see in the door at home is an amazing array of birds of colors and shapes and sizes and very few of them have got to actually eat the rice at any time, let alone during seasonal times in their life cycle. So why do we have wings to scapegoats? Why do they get to the blame of all the loss of yield that we hear so much about? And it's mostly because there is a lack of knowledge out there. There are a few of us who work on those and there are a few of us who go out and have a look. But that information is taken while to get through and so what the big message from this exhibition mostly will be is that all the birds are amazing and that northern advice and that trickles down into a day-by-day kind of situation and farmers realise this. So we can stop the unnecessary persecution of the species which is going on. Now not mentioning that not all birds don't eat rice, they are somehow there and you'll see some of the descriptions by the species behind me of the birds which you need to be aware of and you need to know that there's also a bird in there which can form an ecological engineer system which again, to be mentioned briefly, about how you can use these birds. You can use the birds that turn up and their natural behaviour for taking the invertebrates, for taking pest species and the overall goal, sorry, is to increase yield whilst you're somehow changing the farmer fields for migratory species, for these species which are going to benefit you overall in the long term. So, and I'm just going to tie it up quickly, what can we do next and it's really simple and that's to go through those doors and enjoy every single bird you see in there because they're around, all you need to do is look up and you don't need to be out at 5 o'clock in the morning they are around most of the day, it's just a bit more tricky to see. So, enjoy, see what we can see and learn the species and spread the news that actually birds are our friends and we want to keep them as long as possible. Thank you. Well, under Secretary Serrano is on mission, we are fortunate indeed to have Mrs Serrano join us today to speak a little bit about her husband's birdwatching and her husband's work, please. Claire Finon, Fred, misses this afternoon's activity and on this behalf I would like to leave this message sent through email. Your Excellencies, the Director General, officials and members of the ERIC organization, leaders and constituents of the UPFB and nearby communities, Fran is here with me. We thank you for coming over to Grace's Velvark occasion, probably the first dedicated to birds in the rice fields. We thank and commend ERIC for conceptualizing and initiating this photo exhibit as the organization's contribution to the 30 national year of rice 2013. The value of this initiative by ERIC not just because it showcases the bird photography, the two most active maquilling area bird photographers, Dr. Kirsten Parangis, among my favorite professors from undergraduate days and up to the present still my patient teacher in a lot more good things and myself, but primarily for that informative and educational value to the community. Farmers, students, the science community, ordinary visitors to ERIC, the immediate community and their leaders. It is hoped that this exhibit will foster a better understanding and appreciation of our magnificent Indian friends. From the most common duration pre-sparrow alumbani target for some bird photographers, to the now greatly seen rails such as the water top, lane bush, hand, well there will be aggressive creek. All birds are incredible creatures and their value to any ecosystem far outweigh their perceived prejudice to human economic interests. This exhibit will also dispel the misconception of many, fostered by some as well, that ERIC is a toxic chemical spewing monster of a technology developer. The proliferation of ERIC and beauty algae rice fields of such a wide diversity of birds, resident as well as faithfully returning migrants, is the testimony to the vastly improved environmental friendliness of modern rice technology. It is the ignorance of people of our wildlife protection laws, the environment on the right of every creature to continue to coexist and share this planet with us, that poses the greatest threat to their survival. We hope that this exhibit is just the beginning and with the help of dedicated ERI researchers, like Ms. Chris Meadly, we may yet come up with a cheap and widely circulated handout on rice birds for the general population, especially the young. By my experience, if bird watching is 100% all and wonder, bird photography is about 97% disappointment, given the shooting conditions. But that means Q3% of boundless exaggeration easily overwhelms all difficulties and disappointments, the vindication more than double by opportunities and graces such as this ERI exhibition. Again, my profuse thanks and best wishes to all of you. My thanks to my family for bearing with oftentimes odd behavior, the odd hours, the consistent record of escapes from household chores, and the strange equipment for chases. We hope you all enjoyed the exhibit. Thank you. Professor, excuse the problems. I wish I could call my wife to read my message also. Just some notes. First of all, good afternoon everyone and welcome to this opening of the photo exhibit. My message is really just some words or words of thanks. First, I'd like to thank Bruce for coming up with the idea of having a photo exhibit and following it through until this opening date. There are a lot of people who are involved in this undertaking, some in direct ways, some behind the scenes, and they all have to be thanked for. I would like to especially thank Bruce Paul. Congratulations for orchestrating this event in an excellent manner. Second, I'd like to thank Iri for providing the space within which the birds are able to survive and perfectly themselves. I think that's a testament of the fact that Iri is putting in a lot of chemicals that kill the birds. They're there from one month to another. So thank you for that, because that gives us photographers and bird watchers immense pleasure. Third, I'd like to thank Moses, the Iri Security Office, especially the Glenn Enriquez-type office here, because we have been bothering him every week or so, because we would like to come into their upland and lowland fields, and sometimes I bring visitors from the Wildbird Club Philippines and also the Wildbird Photographers of the Philippines, and Iri has been gracious in allowing us to visit the fields and be able to photograph various birds. Now, why do we photograph birds? Well, you might have read it, you know. It's really a passion, but we do it not only for aesthetic and artistic value, but we would like to think that we're also doing something for posterity. I am part of the Wildbird Club Philippines and Wildbird Photographers of the Philippines, and we espouse what is called conservation photography. And by that, we mean we would like to take pictures, and not only pictures, but beautiful pictures, which are intended to increase public awareness about the presence of many beautiful birds in the Philippines, not just the Mayas that you see. A lot of people say that. I thought the only birds that could be seen here are the Munia and Mayas, but there are in fact a lot of beautiful birds. So that's one. Another thing is we'd like to make known to people that a lot of our birds are critically endangered, and some of them are in fact bad shape. So we would like to be able to sort of stop this kind of development, and we are trying to undertake programs in fact that will help save some of our endemic birds, which are critically endangered. So, in this regard, I'd like to introduce to you our president, the Wildbird Photographers of the Philippines, Ray Santana. And my members, Ben Rojas, and Keith Sunia, they are my special friends. I don't know if my group is here from the Wildbird Mountain Philippines, but I understand that he has been invited. First again, welcome to the opening of our colleagues' exhibit. I hope you will enjoy it. Thank you very much. And now for the last but not least, Paul Warden. Good afternoon, and thank you very much for coming. This is quite an important exhibit to me and to the rest of us, and put in a fair amount of time and effort into this. When I arrived in the Philippines five years ago, I didn't know what to expect when it came to the birds. I knew about the forest birds, and the Philippines is very famous for the number of endemics and the quantity of variety of the birds in the forest. But the farmlands was a little bit of a mystery, and I must admit I had pretty low expectations. And I came out into the field, and to be honest, I was astonished. It was so much richer, so much more variety, so many more birds than I was expecting. In the five years I've been here, I've recorded over 100 species just in the fields alone, which is a truly extraordinary number for farmland anyway, to be honest. And particularly in the Philippines, where there's a lot of human pressures and a lot of persecution of the birds. So this is really an extraordinary place, and a very, very interesting place. Anyway, so all I really wanted to say other than I'm extremely proud to have been a, played a small part in what is I think quite important to exhibit. I'm a teacher, as I think Bruce mentioned. Education is very, very important to me, and I see this as just an extension of that. So I think that's really all I want to say. Just thank you very much. And now we're going to go over to the exhibit. May I invite Dr. Siegler. How about the exhibit? Dr. Paris, please join us here. I shouldn't be the one cutting these guys. And Paul, please. And Richard. And Mrs. Sanama, please. We only have three seconds, so I will defer. Just join the group here, please. And these are... And are we ready for cut? Okay, so you guys, two hands. Two hands at a time. Okay, here we go. You guys after these. Ready? One, two, and cut! Yay! Welcome, please. Bob, would you open the door, please? I know they're missing something.


Jeff Bezos has said his Bezos F fund will spend $2 billion restoring landscapes and transforming food systems. He told the COP26 climate conference in Glasgow that he had gripped nature's fragility when he travelled into space. Entrepreneurs including Mr Bezos have been criticised for spending money on trips into space instead of solving problems on Earth. Amazon has also been criticised by its workers over environmental practices. Speaking to the COP26 conference, Mr Bezos said, quote, In too many parts of the world, nature is already flipping from a carbon sink to a carbon source.  Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


So my talk is about how we at Gojek built a query semantics engine for GoFood to help our users find food faster. So let me start with a little bit about myself. I have a background in physics after which I did a masters in data science, and I've been working as a data scientist for about three years now. I also recently wrote a book called Applied Supervised Learning with Python, and it was published by PACT in April this year. And I'm currently working as a data scientist at Gojek. Just to show off hands, how many of you have heard of Gojek? Okay, that's quite a lot of people. For those of you who haven't, Gojek is Indonesia's first and fastest growing unicorn. And it has grown just 1100% just in the last three years. And now with over 20 plus services within the app and with products across five countries, we are what we call a super app. So one of the services that we have is called GoFood, which is essentially the food delivery platform within the app. And it has over 400,000 restaurants and 16 million dishes on the platform itself. So that's the kind of scale that we operate at in Indonesia and other countries. So let me start with the focus areas of the stock. I'll be talking about how we took advantage of word embeddings to make our search engine more intelligent about some of the data challenges that we faced, including how to find the right data and how we build our training and test data sets, etc. And about how we choose the right metrics for evaluating our models. So the agenda that I have for today is, so first I'll introduce why we need a Quay semantics engine and how it adds value. What was the existing workflow and what was proposed? Then I'll take a little aside into how we represent semantics using embeddings. And then about how we built the Quay semantics engine in, like what are the components that we built using those embeddings? And then finally, I'll show you some of the results and talk about the impact that we had. So let's start with why we need a Quay semantics engine. So the way that we look at search, we break it down into two parts. The first one is retrieving the documents. And this kind of depends on what the search query is and analyzing the search query to figure out what is relevant to the search query. The second is ranking the documents that have been retrieved. And this includes personalization and relevance ranking based post-processing. So in order to give a little bit of context, let's look at what was before we implemented a Quay semantics engine compared to after. So for a search query called IAM, so IAM means chicken in Indonesian. And the reason that we have things like this is because our largest user base is in Indonesia. So most of our data and everything is in the local language. So for a search query IAM over here, we see that there are a list of restaurants and every single one of those restaurants includes the search query in its name. But this may not be the most efficient thing because when you're searching for something like chicken, you're probably looking for a dish and not a restaurant name, right? So what we wanted was besides restaurants which also contain IAM, we also wanted it to contain say, like translations. Restaurants containing whose names contain the word chicken. Along with say restaurants that have been tagged with a category called say chicken and duck or restaurants that include dishes called chicken or dishes called IAM. So this was basically the problem that we had that we defined for ourselves that we wanted to solve is that we were using only elastic searches, native retrieval functions which included exact matches and partial matches along with somewhat fuzzy matches to account for misspellings. What we wanted in addition was an ability to understand the semantics of the query to derive the meaning of what the user was trying to tell us that they wanted. So this is why we wanted to build a query semantics engine. So if we are able to divide the system that understands what the user wants, that means we'll be able to give them what they want with minimal effort on their part. Minimal effort on the user's part means that they will end up being more satisfied, so that leads to higher user satisfaction. And overall means that the user has a better experience. Now, user experience is somewhat qualitative. It's not something we can define. So in order to kind of define how we wanted the user's experience to improve, we had to figure out what was the total addressable market. Now, the total addressable market depends on what is the potential for impact that we have on our user experience. So we look at things like, for example, when you search for something very specific and maybe that restaurant or that dish is not exactly available in your area. Like say, you want chicken biryani and chicken biryani is not available, but maybe mutton biryani is available. Or something like that. So if you search for chicken biryani, you may not get any search results. So in order to surface the user something that is relevant, but maybe not be an exact match, we wanted to be able to understand what the user wanted. So that is one aspect of it. Another aspect would be to make sure that if we are not just looking for exact matches and the user has to say, they try searching for something. They're not able to find it. They try modifying the query slightly. They're still not able to find it. So at the end of doing say multiple searches, they get frustrated and they leave the platform. So they abandoned the third session. So we wanted to reduce that. So in order to measure this, we had to define certain metrics for ourselves. So let's look at what they were. So in order to quantitatively measure improvement, we had to define what relevance was in our consideration. So the most obvious definition or the most obvious way to figure out whether something is relevant or not is to use human judgment. Because when you're looking at a list of results, it's difficult for an algorithm to say that, hey, these results were relevant because the only person who knows what they were actually looking for is the person themselves. So human judgment, since that is not something that we had, we had to use clicks and bookings. So if you're looking at a list of results, if something catches your eye, you click on it. That means it's slightly relevant. If you end up ordering for them from that restaurant, that means it's definitely relevant. So these are the things that we considered as relevance. So for our metrics, the first one that I talked about, session abandonment, how many search sessions led to a booking versus how many search sessions kind of the user dropped out afterwards. Second one, null search rate, which is basically the number of searches which have, say, a very small number of results. So which basically means that the user experience was not as good. So these are the two ways that we kind of quantify our qualitative user experience side of things. Our main business metrics that we were recording were search to click conversions. So how many clicks or did our search result in at least one click? And the second one, the main quality metric that we have is search to booking conversions. How many of our searches resulted in a booking? So now that we have an idea of why we wanted to build a query semantics engine and how we can measure it, we look at what the existing workflow was. So essentially what a user would do, they would type in a query. We'd get the query as a part of a service. We'll send that query directly as is to Elastic Search. And it would return all restaurants which contain the search query in its name. What we wanted was a step in the middle to process the query and extract meaning out of that query. So, and that query understanding bit in the middle is what we called a query semantics engine. So what exactly is inside this query semantics engine? So first, when the original text query comes in, we have a spell correction mechanism as a pre-processing step. After that, we send the spell corrected query to an intent classifier, which determines whether the search query is either a dish, a cuisine, or a restaurant. Then we send it to a query expansion engine, which gives us certain expansion terms which can be say synonyms or translations of the user's search query. And we keep a record of those. And from the intent classes that we got along with the expansion terms, we generate an Elastic Search query, which we then send to Elastic Search. So let's take an example. Can I take questions at the end? So yeah, so let's look at an example. So the original search query was coffee, which is misspelled without any. So the spell correct corrects that word to coffee. And it says, when we send it to the intent classifier, the intent classifier says that say that it was identified as a dish with 67% probability, a restaurant as with 23% probability, and a cuisine with 10% probability. Then so we record these numbers, we find what are the expansion terms. So words that are similar to coffee would be say cappuccino or espresso because let's say if you're looking for, if you want to order coffee from somewhere you're probably open to ordering one of these types, different types of coffee, right? So using these inputs, these intent probabilities as well as the expansion terms along with their level of closeness with the original query. We built the Elastic Search query using the DSL and send that query to Elastic Search. So before we move on to looking at how each of these components is built, we'll talk a little bit about how we represent semantics. So why do we need a numerical representation of text? I think as Katherine mentioned in her talk earlier as well, computers don't understand text. They can only take numbers as an input. And in order to take numbers as input, there needs to be some form of numerical representation. So essentially what we need is a sequence of numbers that can represent each word that we have in a corpus. So it can be any sequence of numbers. One way of doing this is say one hot encoding, which over here there is an example. Say you have two sentences in your entire corpus, which means there are five words, have a good day and great. If you one hot encode them, that means you insert, you create a vector of the size of the vocabulary. And you put a one at each position which represents which word it is. So over here in one hot encoding, we can see that good and great are two orthogonal vectors. And that means that the kind of relationship that good has with great, the degree of similarity would be the same as what say great has with day. Which is not the case because it's not encoding any kind of meaning into this. So what we want is a way to make sure that words with similar meanings occupy close spatial positions in the vector space. So if you look at the same example again, instead of a dimension five vector, we've made a dimension four vector. And good and great would be 0.8 and 0.9 instead of one at different positions. So this indicates that there is a high degree of similarity between these two words. And they are relatively orthogonal compared to the other three words that are there in the corpus. Now what is closeness? How do we define closeness? How do we measure closeness? We measure this using a metric called cosine similarity. So if we have two vectors A and B, the cosine similarity is a measure of the angle between them. So cos theta, if A and B are both unit vectors, A cos theta divided by B is essentially equal to cos theta. And it takes a value between 0 and 1. So the higher the value, the closer to 1 it is. The more similar these two vectors are. And hence, the more similar the words represented by any two these two vectors would be. So cosine similarity is essentially equal to cos theta. So how do we encode semantics into vectors? One way of doing this is by using something called word embeddings. And in order to generate a word embedding, one of the methods that we use is word-to-wek. Now word-to-wek is essentially like a shallow neural network. I won't go into the details of it, but there are two different ways of training a word-to-wek algorithm. One of them is continuous bag of words, and the other one is quibram. So both of them, what they do? They look at a word, and they look at what the context of that word is. What are the other words that exist around that word in a sentence? And they try to see, they try to form co-occurrence relationships. So if there is a word, say, good and great, both of them occur in the context of have a and a, which is how we determine that these two words would be similar words. So for example, one way, one advantage of using word embeddings is that you can form word analogies with these. And you can sort of say that words would follow the laws of vector algebra. So if we have a vector called for fried chicken, and we want to subtract chicken and add duck to it, we get the vector for fried duck. So this is one of the qualities of training a word embedding that we wanted to take advantage of. So now, let's come back to how we build the components. Over here you can see what was the timeline that we had, the workflow. Yeah, the workflow that we had. So we had a spell correction, then the intent classifier, then query expansion, and the elastic search query generation. So first, let's start with what was our pre-processing step, which was spell correction. So we wanted to build a frequency based spell corrector, which consisted of unigrams and bigrams. Which would also take into account the fact like how probable that word was to occur in the corpus itself. So let me explain this using an example. Say the user types in crispy with a K, and crispy with a K exists in the corpus, but it has a count of 1000. And say it exists as a restaurant called crispy creme. Now, there is another word, crispy with a C, which exists 10,000 times in the corpus. Say as one of the examples could be crispy chicken. So what we wanted was that even though crispy with a K exists in the corpus, because crispy with a C has a higher count, we want to correct the crispy with a K to this word. So essentially the output of, if the input word to the spell corrector was crispy with a K, the output should be crispy with a C. So in order for this to work, the user actually should have meant that. And what if the person actually meant crispy with a C? The way that we account for this is in our elastic search query, we still include a fuzzy match clause, which makes sure that even if there's a word within a certain edit distance away, it will still surface results from that search query. Even though that was maybe not what the user may have meant or may have written the wrong query. So that match clause will be given like a slightly lower boost, so that if this query would probably give us more prominent results. But results from this query would still turn out. So one of the alternatives that we had that we chose not to use was this library called simspell. It is extremely fast. It's significantly faster than what we have, but it does not account for errors in the corpus. So a lot of our restaurant names and our dish names are misspelled. So that means that even if a person typed a misspelled query and we wanted a spell corrected, since it may have already existed as a dish name somewhere, it would not get corrected, which is not what we wanted, which is why we wanted something that also took the frequency of the terms into account. So that is why we chose not to use simspell. Next, we have the intent classifier. So the three labels that we use for intent are dish, restaurant, and cuisine. So we know what exactly is the list of dishes in our corpus. We know exactly what is the list of restaurants and we know what are the different cuisines that we have. So essentially we just built a supervised learning data set with the X as the names of these three and the corresponding Y as the label that we know what it is because it exists from our content. We use this library called fast text, which is a Facebook supervised learning library. And essentially what it does is it builds embeddings on the ints, like internally builds embeddings, but it builds character level embeddings, not word level embeddings. So it sort of also takes into account misspelled words and words that do not exist in the corpus. So this is how our prediction kind of worked. So if you have a search query copy, it will return each of the intents, each of the labels with a certain probability. And this probability for each of these labels is what will be plugged into the elastic search query when we finally generate the query. So the challenges that we faced, there are a lot of ambiguous words. For example, burger. If you search for burger, say how many of you would want to actually search for a dish burger? And how many of you would want to search for a restaurant burger king? So it's quite divided. There's almost like equal number of people who are searching for both. So because of this, we know that it exists in our corpus like fairly equally. The algorithm would not give us a very clear output as to what the intent would be. In cases like this, what we would do is we try to enrich our intent by asking the user which one they were looking for. So that we can collect data, feed it back into algorithm, as well as provide the user with more relevant results to what we're looking for. So what were the results of our algorithm? We had about 88% true positives and 84% mean accuracy across our labels. So the next step is query expansion. So what is the purpose of the expansion? It tries to figure out what the user wanted to say rather than what the user actually said. It broadens the query by adding additional tokens in order to include more relevant results and increase the recall. So recall essentially defined as the number of relevant results that were retrieved divided by the total number of relevant results. So the kind of relationships that we wanted as our expansion terms were synonyms. For example, coffee and cappuccino could be considered as synonyms and translations. So ayam which is the Indonesian word for chicken and chicken would be a translation pair. So this is what we wanted to capture using semantics. So like how we actually ended up using fast text for our intent classifier. Over here we decided to build our own word embeddings. So the main challenge that we had was creating the data. Because algorithms like word to work expect you to have like a grammatically correct sentence as an input. But what we had essentially were phrases. We had search queries, we had restaurant names, we had dish names. And we had the names, yeah, so the names of the dishes that the person ordered. Now how do we include all of this? How do we create a sentence out of this so that word to work can kind of learn the kind of relationships that we wanted it to learn? So we kind of iterated over several ways of formatting the data. In our first one we took like for example, for a single booking that a user made. The user searched something, they clicked on a restaurant and ordered dishes from that. So the query word, the name of the brand of the restaurant from where they ordered, along with each of the cart items in one row. So this is what we first tried out as a single sentence. And this didn't really work that well for us. We tried a couple of other things where we included each pair as a separate sentence, on a separate row for a single booking. And in the last iteration, we also included the dish descriptions of each of the cart items. So what we saw was the best performing was the third one, which did not include the brand name at all, but included just the query word along with the cart items. And this was because a majority of people actually end up searching for dishes and not restaurants. So for example, there's this dish ayam goreng, which means fried chicken in Indonesian, which is very, very popular. So ayam goreng is one of our most frequent queries on a daily basis. And it's a dish, it's not a restaurant. So yeah, that is why we thought, that is why we speculated that this was the way that our data kind of like performs the best. The embeddings are the most relevant to each other. So from the embeddings that we trained, we plotted a TS&E plot. TS&E is essentially a algorithm that sort of reduces the dimensions into two dimensions so that you can visualize the embedding easily. So this is some of the more popular queries that we had. So we can see that this cluster over here is different types of noodles that are popular in Indonesia. So that means that the embeddings that we were training. Made sure that different types of noodles had word vectors that were spatially close to each other. Another example of a cluster that we have. Another example of a cluster that we had was this one, which is essentially different types of fish that are very popular in Indonesia. So how did we perform the prediction using these embeddings for query expansion? We took the word like the search query, found the embedding for that word and used the cosine distance to find different embeddings that were closest to the embedding corresponding to the search query. And figure out what were the corresponding words. So let's look at some of the examples that we observed. So bihon and kwetao are two different types of very popular noodles. Again, and these were determined by the algorithm to be expansion terms of each other. Then we had chicken and duck, which were also like, are used interchangeably in a lot of Indonesian dishes. So this was, we were quite pleased with this example as well. Another one was juice and juice, which is the example of a translation expansion pair. And another thing that we saw that we were quite amused by but very pleased with was say an association between different types of similar brands. So McDonald's was the word vector that was closest to the vector for Burger King. Now how do we generate the elastic search query? So I do have some examples of quote, which is at the end, if anyone has questions we can look at it. And the slides will be up, but over here I'll just qualitatively go over. So the intent classes that we had. We saw that there was a probability that was associated with each class. So we used that probability as the boost for different intent classes. So for example, if you searched for KFC, it was classified the highest. Say it was 90% probability a restaurant, 5% ish, 5% restaurant, 5% cuisine. So when you're searching for a match in the restaurant name, that boost would be the highest compared to the boost for the match terms with the cuisine and the dish names. Then query expansion, we added additional match clauses with a diss max function which made sure that if there was a say a restaurant name or a dish name which included both the original word and the expansion term, it would only, it wouldn't kind of increase the match score, it would make it twice as much. It would try to ensure that whatever it matched with most, say for example, if it was a dish called coffee cappuccino. It would ensure that it was, if the match score with coffee was higher, it would only include that match score in the final score. Versus, matching, including, increasing the score based on a match with both coffee and cappuccino. So we used the match terms for the query expansion were added for both the restaurant names, as well as the dish names. And the boost corresponded to, obviously we had the intent boost. But for the expansion terms, the boost was also a function of the cosine similarity between the expansion terms and the original term. So let's look at some of the results and how it manifests into the app. This is again the before and after that we had for the same term, ayam. So over here, we can see that the first result includes ayam in the restaurant name itself. Then the second query, the second result has the word chicken, which was the expand, one of the expansion terms for the word ayam as like a translation. Then the second, third, and fourth results had a cuisine, like if you can see at the back, a cuisine called chicken and duck. So since these results matched the cuisine, there was sort of like boosted up as well, even though these two did not contain either chicken or ayam in the restaurant name. Now the last results, we don't see any obvious matches in the name itself, which means this result was surfaced because it had dishes that were called either ayam or chicken. So let's look at how once we built the square semantic syndrome, how it actually improved performance. We were looking at session abandonment rate. We saw a 10% drop in the sessions that ended up being abandoned. So like a 10% increase in sessions resulting in a booking. The null search rate actually saw a 99% drop. That means 99% of searches that resulted in very few results now had a significant number of results. Search to click conversions, we did not see much of a lift, 8%. But search to booking conversions, we saw a 30% lift. And this was just one of the first versions that we built. We're still working on iteratively improving both the intent classifier as well as a query expansion bit. So some of the future work that we're working on is we're working on trying to add time and geographical context to our expansion terms. I'll take questions once I'm done. And maybe using customer specific booking and search history so that we have expansion terms that are somewhat personalized. Some of the related efforts that we're working on are building a food knowledge graph. At the moment, we're using open source data from TVpedia. And we're also working on building auto-suggest and auto-complete features. So going back to the key takeaways that I mentioned at the beginning of the talk, we talked about how we're using word embeddings to make a search engine intelligent and we're using this in the form of an intent classifier, as well as by using embeddings for query expansion. We talked about some of the data challenges that we face in intent classification because sometimes the query can be misclassified or the intent can be ambiguous. And what or how we try to structure a data to train embeddings for query expansion. And we talked about how we chose our metrics for evaluation based on the total addressable market that we saw. That's it, thank you. If you have any questions on the approach that we took or if you have any suggestions on how we can improve our system, I'd be happy to take those. So, yes. Thank you. And one question that I had was when you talk about search to click or search to convergence, do you look at the unique number of users who searched to unique number of users who clicked or you take instances of search and clicks? Like what is the view here? We take each search independently. It's not user based. It's based on each search session that happened. So each time a user typed in a query, did that result in a booking or not? Okay, so if I search ten times, I would count as ten searches? Yes. Thank you. Thank you, Sita. It was a nice session. Two question I do have.  So once you added those two components, how did you handle the latency part? Because in the middle, there will be more processing. So for the end user, if I just keep typing and the results are not appearing, that is a challenge. And second, which is related to on the spell correction side, I'm not sure which library did you use because natively elastic search doesn't support that. Yeah, so for spell correction, we did not use a library. We included it as a part of our code base itself. Yeah, and your first question was, I'm sorry. Like on the latency, yeah. So the latency did not increase by much, I think about. So our latency limits, I think the 99th percentile is about 200 milliseconds for the search response. And this increased the latency by about 10 to 20 milliseconds because we used a Redis cluster to store the key value pairs for expansion terms and stuff. So it did not really increase the latency by much. But so that being said, this is because currently in our experiments, we have static predictions for expansion terms, et cetera, and intents. So we've taken the top and 50% of most frequently searched queries. And we've built the system for that so that it's easier for us to do A-B testing on control versus experiment. Look. I think. Yes. But for query expansion, you used Word2Vec. So any particular reason why you didn't go for fast text in query classification as well? Oh, sorry. Expansion phase as well. It's because mostly we wanted more control over how we structured our data and the kind of algorithms and the hyperparameters that we used. So we built our intent classifier after query expansion. So we were experimenting with embeddings and how we can use them. And we thought, why not just use embeddings for query expansion instead of representing each restaurant as a doc2vector? So there were a couple of iterations that we went through when we built query expansion. After that, we decided that we wanted to build an intent classifier as well. And since we didn't want to take the spell correct latency into account for that for the intent classifier, we decided to use fast text because it takes care of misspellings very well. That's the reason. It may also help in the query expansion phase as well. That's what I think, anyways.  Because it's unsupervised. Yeah, so some of the pre-experiment validation was mostly to do with human judgment. Remaining, we validated once we experimented with them, we saw the conversion lifts and attributions. So that's how we know that if users are clicking on it, booking on the results from the expansion terms, we know that those are relevant. So yeah. Hi, I have a question. So I'm expecting all those models, the intent classifier, the query expansion, there are supervised models built using neural nets? The query expansion embeddings were not supervised. Those were unsupervised. The intent classifier was a supervised model. What was the measure, actually, where what was the measure you used or the metric you used to measure the model performance? So I think I mentioned this in the beginning as well. For the intent classifier, we looked at true positives. So what we did was we took the top 1,000 queries. We manually labeled them. And yeah, so the top 1,000 queries were roughly about 40% to 45% of the total search volume. So we took those, we manually labeled them, and looked at offline predictions, and then compared them to the actual manual labels to find true positives and the accuracy. Sorry? We did think about that, but we weren't. Yes. But that would work better for a binary classification problem, right? So we had multiple classes. OK, point, point, point noted. Thank you for your suggestion. Hey, this is Jay here, here. So I have two questions. One is from the algorithm, which is the word to wake. Have you considered using LDA to wake because the intent itself is a big problem in the query when the user is giving it? So did you consider LDA to wake? And why did you choose LDA to wake or what to wake? Second question, you spoke about replacing the most occurring word with the misspelled word. When there is a misspelled word, you look for a similar, the correct word and you replace it. What happens the other case if the misspelled word is higher and then the correct word is in lower frequency? How do you handle it? Because I tried to take this as an inspiration. And if I were to use it, I'll take your idea. So we actually did not consider LDA to wake because I have not heard of it before, to be honest. So thank you. I mean, I will go back and read up on this. So thank you for your suggestion. Based for the spell correct, if the misspelled word has a higher frequency. If it has a higher frequency, then the chances are it's not misspelled, and that's a legitimate spelling. So that is the base assumption on which we have our spell correct, because for example, each word has, say, 10 to 15. Each of the more frequent words has more than 10 to 15 misspellings. And those have a significantly less frequency. Say, for example, 1 million compared to 100. So we were pretty sure that if we didn't want, all the misspelled words would probably be removed if we only consider the words within a certain added distance that were of higher frequency. So if they were misspelled and they were higher frequency, that means they do occur in our corpus a lot, and we would want to search those. Can you explain about the knowledge graph, the food knowledge graph, which you are working? So essentially what we are doing at the moment, this is something we actually just started working on a little over a month ago. So we took data from DVP. And DVP data is basically a graph network that's based on top of Wikipedia data. And we ingested it into Neo4j. And we are planning on using that for our next version of query expansion. So basically, based on traversing the graph, what are the most similar words compared to the search query? And yeah, I mean, we're still working on it. Now you have to work on the, from natural language conversion to the Cypher query conversions also. Yeah, so one of the things that we are looking at is basically how to map one, how to map the search query to a node on the graph in the knowledge graph, as well as how to map entities, nodes on the knowledge graph to entities in our content. For example, if there's a node called IAM, or chicken, how do you map that onto the different chicken dishes that exist in our database? So these are two things that we're still currently working on. We don't have a solution yet. But if you have any suggestions, we can catch up after this, I'd be happy to hear them. Hey, regarding the spell, correct. So do you use Peter Novik's algorithm? A modified version of that. OK, so you're saying it would be CPU intensive than memory intensive. Because Sim spell stores all the possible errors or combinations of it in the database and then looks up. So do you see a CPU intensive approach? Yeah, it is CPU intensive. And what are the latencies for this? A distance to the 99th percentile. I don't have the exact numbers. Sorry about that. This one? Yes. Yes. So for example, for the third one, if you have a query, say, for example, a chicken burger, and the cart item that was ordered was, say, a Zinger burger from KFC. So that means chicken burger, if you'd consider center word as Zinger, then chicken and burger would be the context for that. In the first one, the sentence was extremely large. So it varied a lot across different bookings. Because there might be people who order from a KFC and just order Zinger burger. In which case, the sentence would be, say, four words long. There were people who ordered a lot of things from very long name dishes, et cetera, in which I think the maximum sentence length we saw was, say, 250 or something, in the first one. So the window size that we would have to include was very large, which means if there were words that were towards the end, they would not consider the search query, which is in the beginning, as context. Yes. The query went part item maintenance. Yes. So when you're changing that line, what is happening is that the document is not, that means the second line is not, this is the outcome context in the first line. Yes. Yes. So when the window size, the second size will not be necessary. No. So in that case, cart item one and cart item two would be completely different data points. Yes. OK. So no worries. Yeah, definitely. So this works well. Query expansion works best for single terms. If you have multiple terms. So how would query expansion go? We have built embeddings for unigrams as well as bigrams in our dataset. OK. So, but that's the max that we go to. Because we are considering only, so at the moment, say the top and most frequent queries in our experiments and most of them are either one word or two word queries. So that is why we are considering only unigrams and bigrams and building embeddings for those. Yes. Even the boost factor would be with respect to the same. So basically the boost factor is term-specific, right? Based on the synonyms and the expansions. Yes. But the boost factor would need to be generated for the multiple terms as well in that case. Yes. It's there. So in our Redis map that we have, it's a search query along like as a key. And the value would be a list of dictionaries and each dictionary would have the search, the expansion term and the corresponding cosine similarity. OK. Thank you. One last bit of suggestion for autosuggest. We could have trained on queries as well instead of content to get what are, for query expansions, what are the potential candidates for expansions. OK. We can catch up on more of this afterwards. And I think your question, we can go up after this, I think. Thank you, everyone.


This is Witchspace news for Friday the 30th of October 2020, I'm CMDR Buur In this weeks news, the Frontier Halloween in-game event reaches its creepy conclusion, horizons for everyone and more. If you enjoyed this video remember to subscribe to the channel and turn on notifications and if you'd like to support our work you'll also find us on Patreon, links to everything you need are in the description below. As the community team at Frontier had hinted some weeks ago, this week saw the arrival of the Elite Dangerous Halloween event and it was a doozy. As we've been reporting all week the event started with hints of something afoot on social media, cryptic messages in binary code began appearing on the Elite Dangerous Instagram page whilst the game's Twitter account lit up with a mysterious screenshot and a link to the Instagram page. The binary code messages continued over successive days until Wednesday when the full message could be decoded, it read Colsack Nebula, the Stones, he keeps watching the Stones. With the location of something revealed, a large portion of the player base headed off to search the Colsack Nebula for further clues but it wasn't until the game came up on Thursday after an extended scheduled downtime that things started being discovered. We're not going into spoiler territory here in this video but we will say that a Galnet news article appeared in-game detailing the arrival of a long lost Megaship called the Adamaster in the Chuck Chan system that then sent players off on a galactic mystery tour the like of which has not been seen in Elite Dangerous in a long long time. When the event started it brought with it a level of engagement and mass group activity that had been sorely missing whilst it wasn't without some small problems at the time of this recording there are a couple of bugs with the mystery tour that still need to be ironed out, the explosion of frantic player activity that the hunt for the story of the Adamaster caused would surely have proved to frontier as if there were ever any doubt that the player base is still very very active and hungry for more of just about everything in Elite Dangerous. If you've not yet followed up on the events of yesterday in-game the things that have been added are not going anywhere this isn't a week long thing and then it's gone kind of a deal the story that surrounds the Adamaster is very much a permanent thing and fleshes out yet more of the Elite Dangerous lore and is a permanent addition to the existing game so there's plenty of time to experience it yourself when you're ready. Not content with the events in-game however Frontier's community team weren't done with Elite's commanders entirely as the community was wrestling with the mystery of the Adamaster the community team released a hilarious spoof edition of the Super Cruise News broadcast that frankly has to be seen to be believed I'm not kidding it's linked below special mention has to go to community team regular Stephen Block Benedetti for driving for six hours from Scotland to Cambridge yesterday morning to help with the Dave's events and appear in the video alongside Bruce, Vantana Guerrero and Arthur and command told me with the out-of-game hints the news broadcast and then the suitably creepy but still completely in-universe in-game horrors that we think hint at more for the future it was easily one of the best Halloween events we've seen in any online game as we've already said this stuff has added to the permanent elite backstory and lore and is a permanent addition to the game rather than a throwaway week long event so there's plenty of time to experience it all for yourself here at the pit we're still not done with following up all the Halloween in-game content ourselves but with what we've seen already we're happy to report that Frontier have absolutely smashed it wholesale bravo team as was expected this week the horizon's expansion was folded into the main game making way for the Odyssey expansion next year this means that the base game now includes planetary landings on airless worlds access to the engineers ship launched fighters and SRVs multi-crew and more if you already owned horizons then every ship in the game SLFs included now have access to a shimmering new azure paint job and as an added bonus this week the game now with horizons included is currently on sale on steam on the pc for around 75% off meaning in the UK you can pick it up for about five quid and finally this week as part of the Halloween season there are some new bobble heads on sale in the in-game store to festoon your dashboard with creepy delights the horror themed bobbles will only be available during the Halloween season so don't wait to grab them if you're in the market for such a thing that's it for now thanks very much for watching we'll be back later this week with more videos until then 07 commanders follow the greens on the way out and do keep clear of the toast rack we very much look forward to seeing you next time


This is the craziest thing that has ever happened to me. You see my ticket, my boarding pass, I booked the economy class. But when I was coming in here, I saw the captain who watches my videos and he's like, Oh, our favorite village boy, good morning. And that's it, I'm like, oh, good morning. And you know, they upgraded my seat from economy class to business class, man. God damn it, man. Do what you do, don't stop. And my girlfriend will say, nothing is impossible. Make sure to subscribe to Captain and the entire crew of Askeye, when we reach home safely. Thank you, bye.


Nicholas asks, I have a question regarding adding a group to a team. From what I know, you need to be owner of the team that you are adding the group to. Also, users in that group is added one by one, so it is not dynamic. We have several security groups, mail-enabled distributions list, etc., sync to Office 365. What is the best way to do this? The same group is also needed to add as a group permission to a SharePoint site. I would like to use on-prem created type of group sync to Office 365, but what is the best way here? This is a fun one, identity management. I've got to wipe the dust off with this knowledge. You're using on-prem groups. You want to maintain them on-prem? I respect that. It makes things a little bit more complicated. Now, yes, you're right. You can add a group to a team. You have the add members, the wizard dialog box, and it says you can add a distribution list or security group, but that's a point in time. Unpack the membership, puts it in the team, and it's done. If you want that to be a permanent syncing thing, I've only got two options that I can think of. A, set up some automation, power automate, PowerShell script, something like that to regularly sync membership or add the people to the team and cross-reference and remove them when they get removed from the group and so on and so forth, or create a new team and make it a dynamic membership group where it's looking at some sort of attributes on your users and on-prem, instead of adding your users to a group or in addition to adding them to a group, set an attribute for them. So, does this go back to where, in SharePoint, you couldn't add a group as a member of another group? Kind of. Yeah, because you could use security groups, but you couldn't use a permission group or SharePoint permission group as a group within that, correct? So, that's kind of the way I was reading into this is like they're trying to add a group to a group which doesn't work, but I like what your solutions are, Max. Yeah, even worse with the Microsoft 365 groups platform because its members can't be groups, just never can be. Yeah, especially as I'm trying to. The SharePoints that are associated with it, sure, you can screw around with the permissions and they're all you want and add security groups and all sorts of fun stuff, but now, if we're talking about an m65 group, which is a team, you're going to have to do some wonky, custom stuff if you want an on-prem group to in any way sync with the membership there. If you can let go of the on-prem component, your life will be a lot easier, but I understand that there may be other implications there. I love that. It depends. Answer again. All is. Do we answer the full question? I think so. That was. I think it's good to address these things. I wonder how long, what's the life cycle of this? How much longer, when are these people going to move away from holding onto these components where it's not going to make any sense to maintain these on-prem components and this current architecture and update? Well, because they can have their active directory in the cloud and sync to something on-prem, they can make the cloud the primary, right? And then sync it to an on-prem if they need to versus on-prem being the primary and sync to the cloud. This is beyond my scope, I'm asking. You're actually right. An Office 365 group can be reverse-synced on-prem for use on-prem. It'll be overwritten every time it syncs. So if you make changes to the on-prem group, those'll go away, every sync which happens every 30 minutes unless you customize it. But yeah, that's a great point is if you want to reverse the way you do things, if you need that group on-prem for some sort of on-prem app, that's an absolutely good idea of a way to go. I just ventured into my danger zone. Yeah, but it was a good point.