e different requirements of the edge machine learning we have divided that into three parts so the machine learning requirements the edge computing requirements and then you have some overall requirements that are related to everything from the machine learning perspective you have the low task latency the high performance we need always high performance to when it comes to the computation generalization we always again we need some generalization when we find our models enhanced privacy and security and then we have to be independent from the data that is labeled from the edge computing we have the efficiency in the computation optimized bandwidth offline capability and low communication latency this offline capability comes most of the time because the people they may have issues with the network and they still need to use their services that's a very important thing and this should be allowed by the edge machine learning and you have the cost and the energy that I would say is related to all so we have to play with all these parameters when we think of doing edge machine learning it's not an unlimited resource as we have in the cloud for example so you have three parts when it comes to edge machine learning you have the learning on the edge you have the inference on the edge and then we have some parts some of the work that is done around the preparation of the data directly on the edge so the paper I was referring to is this one for those who are interested it was published few weeks back it's a long paper it's around 50 pages but we go through the different aspects related to edge machine learning so from the inference I try to summarize things in this slide we have different ways of applying or bringing sort of a big model to run on a smaller device so the first one I believe everybody is familiar with is the quantization we are working a lot on that so we are trying actually to work on the models to change the encoding of the data to reduce the encoding of the data so that the model becomes smaller but still we keep the quality of the model we are able to quantize models up to 4 bits from 16 bits and we keep a very similar sort of performance you have people who are working on the weight reduction we are not sort of handling this part for the moment but the idea here is to see it's a sort of approving strategy that we are sort of exploring we have the knowledge distillation the activation approximation I'm just going quickly on this one while we are not doing much on that but on the last two ones the early exit we are exploring it a lot combining it with the model compression and we have the caching when it comes to the exploitation of the large language models on the web so the inference on the edge has a reflection I think you have a variety of methods that are there promising results you have sort of methods that are able to reduce the models 32 times and then bring them with a very similar quality you have lack of real public implementations we don't have much of the things that are stable and that can be commercialized for example which means we still have a lot of work to be done at this level lack of automation to do this quantization for example you need to have people who will be working basically it's a try and fail so you see what is not working so you try again again and again until it works this is also related to the diversity of physical architecture it depends on your processor it depends on the physical capacity that you have so you need to have people who should be behind that to control it again we are investing a lot on this part trying to bring some solutions to automate the quantization because it's very important and again it should bring an answer to all those businesses who do not want to see their data moving outside their company we don't have a killer application yet so more work is needed for those who want to work there there is really a lot of effort that need to be spent learning on the edge so that this started lately basically you have different approaches again but the objective of what we want to do is to build directly the model on the edge right so we don't want the help of the cloud we want to build directly on the edge you have limited constraints and you need to work on that you have different methods I'm listing them here the most used ones I would say is the distributed learning we are trying to distribute the learning on different devices everybody is aware of the I guess the federated learning but you have other methods also that are in the same space I would say we are exploring it but here we need we are exploring it from a theoretical perspective in the theory team but when it comes to the application or the concrete sort of deployment of these kind of things we get a lot of physical constraints again that need to be solved so we are trying to work on that especially when you use for example heterogeneous devices that's not an easy task to have where in theory for example we try to sort of make all those constraints basically we don't take into consideration that so we have the federated learning we have the split learning we are also exploring transfer learning again I will share the slides for those who are interested just not to go into the technical things but then we have the summary that we have tried to build on the Edge ML so this is the taxonomy again we go back to the reference edge learning and data preprocessing you have a lot of methods that are there but again so in terms of papers, research papers we have a lot of papers that are there I think the paper we have published has more than 250 references but when it comes to the industry and the platforms that support those kind of things this is very limited and I think there is a need for investment to make things happen because again this will help bringing these LLMs to be built or at least the inference could be done at a lower cost to generalize the use of these LLMs so this is just sort of a summary on the learning part so the generalization and the adaptation is complicated to do when you do the learning on the Edge while we need to set the theoretical foundations of that the architectures we are always facing the issue of heterogeneous devices so this is an important thing that I believe needs to be taken into consideration when we build that theoretical foundation you have the sort of the hybrid approach that should be exploited in my opinion hybrid in the sense that I can use the Edge but I can also use the cloud as a collaboration between these two things data quality and assurance any business you talk with will tell you that I have issues with this because again we assume a lot that the data is good but in reality the data is not that good so we need again to find sort of methods that would verify the quality of the data we have a PhD thesis that is working on that and then you have the standardization that needs to come again to help us into this diversity of devices so I brought like three use cases here just to show you quickly some of the work that we are doing in the in the Edge site so this is George who is trying to show us something so what we are trying to do in this application actually is to show the possibility of building an Edge a model, a machine learning model, a neural network directly on the phone we illustrate it on the activity monitoring so what we are trying, what George is trying to have here, he has a small model that is running on his phone and he just checked or showed that the working is we can identify it and then he just tried a new sort of activity that is not known by the model so what he will be doing now is to just record some samples of data of that activity and then he will rebuild the model again or update the model directly on the Edge so we are here he is basically collecting some samples of data and shortly he will run now he got tired so now he can run the training the thing that is important here to keep in mind is that the training is happening on the phone we are not sending any data to the Edge, to the cloud but everything is happening on the phone it is taking a little bit of time the idea was not the performance but to show that there is this capability that we can build and update the model directly on the Edge so now the model has been updated now he will test basically he will start the inference and see if the new sort of data has been added to the model or if the model has been updated or not and then he shows that and I think those who work a lot on neural networks we have this issue of catastrophic forgetting as you learn the new things you forget the previous things we have integrated that also in the system we are able to incrementally update the model while not forgetting the things that are that have been already learned so the second case if I can go is the sort of reinforcement learning algorithms that we are using to help in the navigation of the drones we are trying actually here to combine image processing with reinforcement learning the idea is to help the drone to autonomously explore an area without getting or without colliding with the obstacles that you have the final objective is to run of course these models directly on the Edge on the device that is the drone the nice thing here again is that we learn some situations it's reinforcement learning strategy then the environment can change but the model is still working properly and the drone will necessarily prevent getting into collisions against the worse I just jump quickly to the to the situation where we add it's this one normally so we have added some obstacles you see the black obstacles are new they were not in the these ones they were not in the initial learning environment but the system is able to identify or to recognize at least the obstacles and go further the next one and the last one is the web navigation so here what we are trying to do actually is to integrate these models in the web navigation so we have the sort of extensions of the of the browsers that you use and any page that you visit you can actually request for summarization you can also discuss with the page or sort of interact with the page with questions and answers this can be working with the cloud but again what we wanted to do is to bring these things to run on the edge so what we have done the work that we do on the quantization we quantized the models and then we brought them to run on CPUs and then we add them to when you get this extension you get actually an instance of the model that comes on your computer so whenever you discuss with any page on the web your data stays on the edge and it doesn't go it doesn't go outside so here we have asked for the summary and then we started asking questions on the content that is on the web page so here for example how much funding is allocated toward low carbon solutions and then with some after some time you should get an answer explaining that we have done different questions and then the system is getting the data from the page and solve all the problems but they should be also mixed with what we call traditional machine learning models and then you can have sort of hierarchies of model coordination of models as we do in web services for example and then have a sort of more complex system that would bring solution for more complex program and more adapted situation than what we have nowadays so the future generative AI should be multi-model that is an important thing to keep in mind we do not want only text or only images we need combination of different types of data specialized models should be at the heart of the ecosystem again we don't want huge models that are too generic or at least not only we need to build collaborative strategies between all those models we need reasoning capabilities so that we do not have nowadays and then we have security and privacy that needs to be taken into consideration and the last one and I think it's the most important one the actionability that needs to be attached to the generative models as of today the models are only generating data so they can recommend things for you for example things to do things to watch, things whatever but then when it comes to the action you have to do the action you have to follow up and do the real work there is a need to integrate some actionability in these models to make them really supportive for the end user and for the business I think that's all from my side thank you I don't need to run I have a question now some phones starting to have some accelerators for specifically this computation for example neural engine in iPhones the question is how accessible they are and what is the perspective because as I understand now they are not accessible like neural engine etc what do you think about perspective in the next three years that's a good question thanks Kiril so I think these things have started always in the same I would say pattern so the device is always expensive but as we move there are new technologies that are coming and they should be accessible but to be fair the people who are able to again it's matter of how much funding people have so the people who are able to sort of budget things for 4000 GPUs for 6 months I think that's accessible for us so we started actually acquiring some of those devices trying to sort of play with them at least for the moment and see what we can do but I think those will be generalized soon with the new sort of architectures that we are having you can check like NVIDIA you have different companies that are trying to come up with new devices with new architectures so in my opinion that would come accessible much faster than we think could AI be used in genetic engineering and nano robotics if yes how yeah that's a good question well nano robotics well not at that level I believe I mean we need to explore it's a matter of how much computation we sort of we can exploit so we need to look into that I don't have a sort of direct answer to that I need to explore a little bit more my understanding is that we don't have a lot of computation at that levels we need to look into that that would be my answer for the moment because you are taking me really like much lower in terms of in terms of edge right so we need to think about that thank you Hakim any more questions so as far as I can get what edge AI is about is that the thing that's computed on the user's devices that's what the user asks for but is there any room for exploration about computing everything like on the network of devices that are kind of signed in to use the service like something similar to torrent network yes there could be issues with privacy but maybe ciphering the data would help with something yes definitely I mean the edge alone I don't think that it will solve again all the issues right so I think there should be a collaborative approach where the edge is collaborating I mean with the small devices they collaborating together as they need also to collaborate with the cloud so what the message here is not saying that we should eliminate the cloud from the equation but it's more to collaborate devices they have to collaborate together but they still need to collaborate with the cloud because at the end of the day you may have some staff that need to be collected at the cloud level and then computed there but there could be a peer-to-peer that's in terms of protocol you can have a peer-to-peer approach where different sort of devices are collaborating to build something thank you and thank you for the talk thank you thank you any more questions okay so maybe related to previous question what do you think about the agent approach to LLMs where you don't have this one giant big boss LLM but rather have a protocol of communication between different LLMs and maybe some specialized LLMs know how to better answer such question and others know how to answer question in another domain or in another language and related similar to human society there is certain distributed system is this model falls into this edge competition paradigm or that's probably different well I totally agree on that I'm really aligned to that and that's why I try to build a figure that we have here so you will have the models what you call agent is basically a model so the models they have to collaborate between them but then this collaboration can happen one-to-one for example one model and another one but you could also use some more classical machine learning things because the generative model will not solve everything but then you will have other layers also that will be coordinating sort of more complex or building more complex sort of logic that needs to be executed thanks but what human language is the most complex system ever invented right and this is actually a mechanism to communicate but for LLMs for artificially created systems what should be this protocol do you have any idea should be this also human language or should be its artificial language what should be efficient protocol to perform this communication like an interface I would say that's an open question we need to work a little bit more on that thank you well it seems that we have no more questions still have someone I have not seen properly go ahead I wanted to ask are you concentrated on general tools and technologies or you also work with requests from companies about their precise tasks that's a good question I would answer to this question in terms of organization that we have so from the TII side we are more in that R&D basically building those generic tools or optimizing working on this edge for example but then we have another entity that is called Adventure One that is more focusing on interacting with the customers than sort of doing some fine tuning and this kind of stuff so let's say a company may come to you with their request and you will tell them what you can do to optimize their processes okay thank you I'm making a pause to be sure I think we are good yeah so okay let us thank Hakim again thank you and now almost immediately we have like in five minutes we have started the next session so I'm reminding that we have two parallel sessions and the one on theoretical learning and optimization will be in a different room and those who have O&OP they stay here start O&OP session session on machine learning and data analysis on the different whole room let's start with the first talk thanks good afternoon everyone it is my pleasure to present a work on multi-label topic classification for the language, our joint work with Sergei Nikolenko of the Stake Love Mathematical Institute in St. Petersburg and other places and Gulnara Kabaeva of Kyrgyz State Technical University named Dr. Arzakov a little bit of introduction and motivation Kyrgyz language is one of the languages of the Turkic family or the Kipchak branch and several millions of people can call it their mother tongue mainly of course in Kyrgyz but also in China, Tajikistan Pakistan, Uzbekistan and Afghanistan and Russia however and despite of certain corpus of research work with computational linguistics flavor dedicated to Kyrgyz language the number of open language resources for Kyrgyz is rather small and upon trying to solve some applied problem that's that involves usage of Kyrgyz language one often meets with certain obstacles due to the lack of language resources so one can say that language is definitely a low resource one until recently and here we can say thanks to Zberbank there was no general purpose LMS for Kyrgyz however as for many other languages from those hundreds there are two families of models that were trained in multiple languages on common crawl and other large bodies of text one of these hundreds of languages is Kyrgyz so one can attempt to use Excel and Roberta Base Large etc Roberta Base Multilingual Cased and as we've just discussed although certain solutions may arise in the nearest years the current NLP still leans towards universal models but still despite that a reliable evaluation for any language is necessary and arguably is impossible without manually annotated resources which is why we decided to develop to make a first step to develop the first dataset for an applied task which could be suitable for fine-tuning some LLMs for the applied Kyrgyz task to just find out whether it is at all possible the answer to this question is not as evident as one may believe and after that we will publish a benchmark our competition is ahead so we haven't yet but we'll publish a benchmark with all the results all the models and all the data so we've built our own dataset with the kind permission of the editors of the 24K GE agencies that's the Kyrgyz news agency we've scraped the Kyrgyz language section of the site that yielded 23,000 news articles and on that site there were no topical texts for news and Kyrgyz so as you can see at the bottom of the slide certain topics are present but only Russian texts are annotated with those and actually they are not quite suitable they are too general in some of those so to say topics are actually multi-topic projects so we had to decide on certain labels to do that one could try to use some general purpose thing like they use in advertisement like IAB taxonomy Demos taxonomy in the older days but those are also too broad we have attempted some zero-shot approaches and unfortunately nothing worked and in certain private conversations with practitioners I was told many times that when working with topical data from a single source it unfortunately has to be custom so that's what we focused on but one can't just invent those labels and we decided to do the so called exploratory annotation to do that we've sampled 500 texts we've translated all the titles into English applied obtained sentence-bearings for all of them and grouped them by hierarchical clustering and then the manual annotation for every cluster we've attempted to invent a title which is general enough to cover most of the news in the cluster and then add some extra labels for multi-topic texts so after that quick introduction we did the re-annotation again from the start because say the second half of the data some labels that we've introduced there were not available at the start so that's the example of the cluster and the proposed annotation clearly all the texts are about certain finds and crimes but sometimes with the political flavor sometimes with ecological flavor and so on having done that we've decided that the label set is established and then we've done the same thing with two 500 text batches which yielded a dataset of 1,500 texts so the first two batches were then used as a training set 500 plus 500 were used as a test set and here's the statistics here are the label statistics for the first two batches one can see that well arguably the distributions of labels are relatively similar which can also arguably signify that the annotation procedure and the annotation scheme were relatively consistent in the process so hopefully the label this yielded 20 topics as for experimental setup well it was pretty standard but with a multi-label multi-label twist we had to do an accurate splitting so that the distributions of the labels in the splits would be similar so we did iterative certification for bag of angrioms models with two-fold cross-validation because the dataset is rather small for neural approaches which are computationally harder we've used a simple trained dev split but basically the same splitting procedure for statisticalization we've used an LTK and we are lucky to have a morphological analyzer for Kyrgyz language from the project Apurtium and that we've used for something that one could call stemming and basic word tokenization solely because I guess one could use an LTK or something or something maybe from the standard package but this is something that seems to be more or less standard for Kyrgyz okay now for the models so we are about to build a benchmark and we've tried some very basic bag of angrioms models with an extensive hyperparameter tuning and those were two groups of methods one group is based on linear models and well our own logistic regression and on stochastic gradient descent with well basically on basically on linear SVM and logistic regression the first approach that we call independent classifiers this is just a set of independent binary classifiers for each labels and the chain approach is also standard approach with classifiers that are trained in a row and the dates for training for the next one in the sequence uses predictions of the previous one the other approaches may not be a great choice for large sparse vectors but we've included them since they are truly multi-label and yet classical enough for the nearest neighbors flavor for multi-label classification and here go the results so for evaluation we've used several metrics probably the most descriptive one is sample wise zhikhar index computed for every pair of predicted labels and sets of goal standard labels and averaged overall samples so the numbers show that the performance is not that great overall also we've computed exact measures hamming distance F measures the micro averaging flavor and also sample averaged version we have also added the count of the percentage of times when at least one label was guessed correctly what we see here the first approach is a simple bag of row talking engrams with different hyperparameters so these are the best groups of hyperparameters that we found with grid search for each family and the results are not great and clearly as expected the nearest neighbors methods perform poorly and when we move from row tokens to character engrams also quite expectedly we get a boost in performance when we move to stem tokens we also if you will just compare the numbers here and here and to the stem tokens we also get an improvement also quite expectedly since the language is morphological reach, glutinative language and removing almost all effects gives a boost an interesting observation is that when we take stems and and also convert them to character engrams this also allows to improve the results and that's it with the very basic approaches but probably the most important thing we were planning to do with this benchmark apart from publishing it is trying some fine tuning some multilingual language model we miserably failed with training birth multilingual so it just couldn't produce any reasonable results however hard we tried but we've achieved certain success with Roberto Lodge with BPE tokenization and as you see it outperforms all the previous approaches by a large margin it is also important to note that for bag of engrams bag of stems approaches we did quite an extensive hyperparameter sweep, hyperparameters you may have noticed that this annotation approach this size of the data sets and everything else is not without flaws and we're going to add some more new evaluations to the benchmark first of all we'll try to apply the model that has appeared before the revised deadline the MGPT Kyrgyz it is also and as the reviewers rightly noted that it would also be nice to translate the Kyrgyz texts automatically into English and apply some English based models like birth or something just fine tuning to test how well this is clearly something that should be done for low resource languages to test what can be achieved by state of the art also we believe that zero shot learning prompt engineering approaches which were not quite successful when we tried them should not be thrown away and maybe we should return to that to a certain articles conversion into something for prompts but most importantly I guess and due to certain recent developments in Bishkek that is kind of easier now we will annotate more data with the help of multiple native speakers and we will do it properly with the instruction with a fixed instruction and several notators for sample just since I have some time just a short note this work was carried out mostly at least the annotations were carried out mostly in 2022 but something changed in 2023 large community evolved large data science community evolved large sub community of volunteers appeared in Bishkek and we've done something else some named entity recognition data sets and some more efforts are on the way such as yet another Kyrgyz corpus so there is more work that lies ahead and it of course should be done whether or not without our participation I think that's it thank you for your attention so thanks a lot of work has been done and one question which I'm interested in is whether transfer learning strategy through multilingual language models is preferable over just pure translation of data sets let's say you don't have resources for Kyrgyz you have a machine I think you have a machine translation system one way is to just blindly take and translate the data sets and train something and you will and you train a model specific language as a Kyrgyz MGPT you get certain quality or another way is you get a multilingual model which supports a lot of languages you train on let's say English data set you assume that somewhere in the model there is a separation of knowledge some model learns to let's say detect sentiment positive negative sentiment but then it transferred to Kyrgyz so in practice if you just need to choose which would be in your opinion the working better out of the box thanks for the question let me just clarify the first part was about translating something into Kyrgyz yes that's actually a great idea and that's something that colleagues in Kyrgyzstan are doing for other tasks but I would also like to ask you to clarify the last part of the question the last part would be not to translate to assume that translation is prone to errors and instead you train on a clean data set let's say for English and not monolingual model but multilingual model which just was pre-trained on Kyrgyz so it knows that Kyrgyz exists but it trained on some English data set or Russian data sets and then it starts to solve tasks in Kyrgyz and then the question is which is more preferable strategy to translate with errors but still learn on monolingual model for the target language without or rather go this knowledge transfer approach thank you for the question yet again it's a good question and I don't have a good answer unfortunately I have beliefs only for that first thing I've met what I've met the thing with training on English and shared knowledge within the multilingual model we did that with information extraction but for my experience on other tasks of the sorts with Russian language mostly I believe that the first option is more preferable so translating something some standard data set if possible to Kyrgyz and fine tuning on it should be better I think based on my experience on the tasks that are slightly less relevant than they could be that's also kind of my experience it's a very strong baseline but I just wanted to thank you for your talk I have several questions the first one just an idea because normally if you don't have a lot of data set on some task you search for another language that is quite close to this one so do you know which language are close to Kyrgyz are there models related to that so maybe in future work you could rely not only to the multilingual ones not only to translation so maybe using some Russian, Belarusian, Ukrainian probably so they are related and you can probably tune some models so what's your experience in this case thank you for the question speaking of the similar tasks in similar languages I've seen speaking of this particular task text classification even without the multi-label twist what I found to the best of my knowledge there are some data sets in Turkish and in Kyrgyz written in Arabic script the Chinese flavor of Kyrgyz and and nothing else that I found but speaking of translations into similar languages there is a stat work of 2020 by the Turkish inter-lingual community which is now I guess special under group in Turkish languages of ACL their model could be used probably but for the task but I really don't see any data sets of the sort in other languages but of course the processing community does exist the in Kyrgyz is rather scarce in terms of resources but the situation is a bit different with of course Turkish language, Uzbek language but for this task I haven't found anything else okay thank you and one more quick question your data set that you are creating is very specific so it's multi-label and it's in Kyrgyz you told that you're going to reuse it during some for benchmarks for Kyrgyz or some other things like correct me if I won but are there any reasons why this specific data set multi-label because there are many other tasks that you could start with were there another specific reason for some application maybe or what was okay thanks but well I would say that okay there was not any specific motivation or inspiration for that apart from the fact that in my opinion and vision topic classification is one of the tasks that average data scientists could meet in one's practice so something like things like sentiment analysis or topic classification I don't know maybe that's it I mean named into recognition is already something on a different level so that was the motivation and it is like one of the classic tasks of information retrieval so I just decided that that was a good idea thank you very much sorry if I missed something in the beginning but can you elaborate a bit about the annotation quality check how do you guarantee that how many annotators do you use some crowd sourcing or do you have plans for this yeah the pretty descriptive thanks for the question the pretty descriptive phrase in the beginning was quick and dirty so that's it so I was actually planning at some point well I have a certain access to experts that are more proficient in Kyrgyz language to native speakers that would check yet again I mean one of the authors is from Kyrgyzstan but still so yeah the only more or less sophisticated procedure for establishing something guaranteeing some quality was the one I have described on choosing the label set as for the quality quality check that was just reading stuff multiple times thank you for the work anyway it's a lot of work okay so there is a remote so I start straight with a brief introduction to the problem the summarization there is two approaches extractive leverages existing text fragments to select a set of highlights and obstructive summarization improves on an extractive by employing additional language resources to paraphrase and combine the set fragments into concise sentences now the main approach to solve obstructive summarization is to sequence where we have encoder to extract the contextual information and the encoder that generates the summary in accordance to this contextual information and the preference is justified since GPT models that have several times more parameters just fail to achieve the level of performance specialized encoder decoder or counterparts especially even after fine tuning as was proved on the second plot that's from official open AI article about summarization so they proved that T5 performs better in terms of human evaluation then they are fine tuned to GPT-3 and now there is an evidence that classic sequence-to-sequence approach is not enough several works just showed that integrating extractive summarization in training and inference in loop improves the quality substantially especially for full transformer models let me the point of works so what distinguishes full transformer models from other architectures is that besides encoder-decoder bridging the attention is used for all intermediate embeddings in all layers meaning that the overall impact of attention is much larger to an extent that attention patterns are now part of summarization models so the models that are more aligned with ground truth extractive labels happen to perform better, converge faster to more optimal results since they spend less time searching for important sentences and just learn to paraphrase and combine them so many researchers argued that it could be beneficial to correct this attention and the first approach is to use the local mechanism binary masking of attention mechanism so it works by just selecting the important parts of the sequence and the problem is that it's equivalent of token removal meaning that the model wouldn't not attend to the masked parts so the information won't be propagated and mainly the whole information the centrality of the context would be shifted in the other direction meaning that the optimal summary would be different so to alleviate the issue the researchers came up with an idea to apply the content selection masking to a subset of layers and attention head they do so by searching for responsible for silencing evaluation and applying the mask obtained from some content selector, maybe an extractive summarization system, maybe some query from the user and the alternative, the only existing alternative to that approach is just complementing the existing attention mechanism to receive some more complex guidance signals with relevant attention this is the latest in state-of-the-art approach uses semantic query document metrics and applies some simple linear transformations and aligns it using some simple interpolation with cross-attention weights to guide the decoder to query relevant positions and we hypothesize that actually there is no benefit in tampering with existing attention mechanism because it still interferes with natural information flow and for alternative solution we looked for inspiration in image processing area namely text-to-image model dali model uses one interesting technique it has no name they call it just result blending it is based on the idea that the model uses click embeddings that maps both text and image embeddings into the same vector space meaning that if we take two different prompts two different text sequences encode them and then just take a weighted average we will obtain some intermediate image embedding and it seems that the result is quite stable so following the same idea we derived biased encoder mixture it is quite simple we don't use different inputs we just use different attention masks one full attention mask go through original encoder using the original input and then we derive selected mask and process it we can use the original encoder or an auxiliary it is just our expansion the theory is that if we use it is more sensitive to masking that is more focused on masked positions it would provide more amplified signals that would better guide the original embedding to very relevant positions and so the decoder would produce more optimal results so to test the method we derived to masking strategies first is based on just using ground truth extractive label statistics and the second one is dynamic in case if we are planning to use it in practice it is based on extractive summarization system just any kind we just need some distribution over sentences that would denote the silency of the set position and so to obtain the mask we just use top piece something of these sentence position distributions we evaluated the methods on four domains well no, three domains just one is represented with two datasets so news, science and dialogue we used the respective state of the art models to determine everything for the methods we used simple grid search on validation part and quantitative results are quite promising biasing code mixture seems to outperform every attention modulation method in almost every scenario and in best case scenario biasing code mixture can bring 8% improvement over the original and in terms of quality well it was important questions so how it performs does it violate the coherency, the relevance and it seems that none of the methods violate these constraints there is a sampling from the news dataset so there is a reference the original model prediction and then there is a set of alterations corrections so relevance attention is based on semantic similarity matrix it just injects the named entities into the original generation attention does the contrary it deletes some excessive entities like there was China and other nations now it's just China and the UK is now replaced with official say which is quite questionable and biasing code mixture is the most different of the bunch it just simply forces the model to revise the whole summary since it creates a new embedding the model understands that text is likely to be different the new summary tells us the story about submarine drones that can walk independently and track over thousands of miles and basically biasing code mixture version is more aligned with the reference summary and so about the patterns, about alteration patterns they are also different in well the original generation can disagree with the reference at any position meaning that the generated summary can be quite different from the reference so the attention based methods don't take into account the initial mistakes from the initial positions however they scale with the article length so closer to the ending they introduce the changes and the attention is just guaranteed to revise the ending biasing code mixture is more radical and more uniform yet still a bi-model it is more aligned of course with the intended reference distribution and as we've seen before it can completely force the model to completely revise the generated summary the patterns of semantics well the attention based are quite conservative the relevance attention since it uses semantic similarity matrix just keeps everything in check with the original generation their attention is more brave and just has a wider range of semantic changes and the biasing code mixture is bravest I would say it just can completely diverge from the original meaning yet it's still better aligned with the intended extractive summary so this concludes my presentation I'd be glad to answer your questions well you're referring to Goose data sets from multilingual or the one that was collected by Goose but they are quite noisy the reason why I chose CNN-DLML for ablation or for case study is because it is the data set that was proved to have it was written by editors the summaries were written by editors so they are quite aligned the Goose used automatically extracted summaries and sometimes they filled with just some automated things so they aren't so reliable for the experiments and besides there are no just counterparts like I used Brio, Pegasus models and yeah Brio, Pegasus but so it's summarization specialized models they are just none for the Russian thank you I have a question it's more like theoretical question do you think your model or am I true that the data set you are using it's summarizing just only one article like not multiple summarization multiple of multiple articles right? do you think it would be possible to transfer this summarization to like multiple article summarization and like applicable or not or what could be the difficulties to do this you're asking for technique well of course one of the main approaches to solve multi-document summarization it just pretend it is just like one long document with multiple chapters so just using some special markers some special tokens we can produce some special embeddings for these chapters and so it's distinguishable from the same long synchronous and yes of course bias encoder mixture can be applied to any model that has encoder so even if you have multiple encoders if you well that's one of the approaches that's just to use an individual encoder for each part for each document then yes you can still apply the same approach for multi-document yes thank you because normally the problem of multi-document summarization is that if you just make them as one long document normally the model looks at the beginning of the text and of the end of the text because it contains like the most summarized information that the model should look into that's why sometimes at some experiments I have seen it doesn't work then you just combine all the sentences of the text in one large text that's why I was thinking yeah so how it could be yes the context window just illimited and the attention patterns just get lost so it could be now I recall yes it was one of the hardest work before transformers and they really did they used multiple encoders and just interpolated their embeddings well using of course not just simple interpolation they used fully connected layer to just to combine these embeddings into one and just passed them to decoder and they proved they said it worked better than just you passing the long sequence but of course it was the recurrent neural networks now we have GPTs and so they claim they can accept the context of up to a thousand tokens yes so just don't know never tested thank you thank you my name is Ekaterina Zalivina and today I present my work of Dementic Detection of Dialect of Pictures of Sculpt Dialects and Speech of Native Speakers the purpose of my research is to create a model for transcribing dialect speech in this work we focus on Sculpt Dialects and provide researchers with a tool to detect dialect of features characteristic of these dialects in the speech of informants why it is important as a field researcher I know firsthand that field data is collected manually so our tool allows to reduce the amount of manual work and concentrate on analyzing linguistic phenomena and we present experiments using Russian dialect data which are not so common now in solving the problem of automatic speech recognition what about the steps first we collect corpus data then for manifest for automatic speech recognition task and do the manual annotation for detecting features we fine tune models for speech recognition and have three approaches for detecting any features and we choose the best approaches and create a big pipeline to work with them what was the data data in the corpus oh no this one we must mention that was used to corpora were taken and the data was collected during expeditions to Red Region and Pskov Region and on the map you can see the location of the villages where the data was collected and it is worth noting that the Pochetsky villages are located closer to the borders of Latvia and Belarus although these dialects belong to Pskov dialects and existing dialect specifications note a lot of differences between them how we process audio data we take files from corpus if files or text-read files we export annotation to text-read files to have the similar files and then we have an audio segmentation based on sentence length we clean from and convert all to lowercase and we generate the file in format that presented on the slide and you can see a little bit of statistics that we achieved and I can say that these very low-resource dialects and we work with a little amount of data and about text data notation for binary classification for each annotation we assigned one if it demonstrated the implementation one or more dialect features otherwise zero and for rule-based and token classification approaches each token in the notation was assigned in the IOB2 notation where B is the beginning of the sequence, I is the second or subsequent tokens for the sequence and O is the absence of a dialect feature and on the slide it's presented the sample of my picture about phytonine approaches it was four approaches and we can see that we have two corpus and for first approach we train on first corpus and then test on the same corpus and the same approach is for second corpus and third and fourth approaches we have two iterations of phytonine first one we you know phytonine on Zapodon-Winsky data and then on a Pochetski data and test on Zapodon-Winsky data and first we train the same way and test on Pochetski data we use the base matrix for this task for speech recognition is word error rate and character error rate and for dialect features detection precision recall of one score and accuracy speech recognition we select models that were pre-trained on standard Russian we use the three architectures among the common mistakes of the two first models are combining two tokens into one splicing one tokens in two parts inserting characters and to correct such errors we use a spell checker Yandex power and we see results on this fight and we see that for Zapodon-Winsky data the best model is the first when we train on this data and test on this but for Pochetski data the best result is a first approach when we fine tune on Zapodon-Winsky data then on a Pochetski data and we get better results for detection of dialect features we use three approaches binary classification of entire sentence binary classification of token and multiclass classification of each token as for a rule-based approach we took a list of phonetic morphological and syntactic dialect features from grammatical sketch then for each rule we write over the function for example to determine the realization of a name in the first prestressive model we follow this algorithm we use the dictionary partial library to obtain transcriptions instead on Russian then we determine the backness and the height of the prestressed vowel and the presence of polarized consonant before the vowel and for morphological and syntactic features we use pymorphy 2 and Natasha we know that this approach successfully handles correct identification after detection for dialect features at the level of phonetics and morphology but variability is not taken into account and all positions in which dialect feature can be realized are marked with an attack but we know that now it is very common to have variability in dialects about entire audio classification for each audio fragment more frequency capstone case and work calculated and during the experiments three classifiers were used and as a result we see that the method is more peaceful rather as one of the intermediate stages but the problem of classifying tokens within a fragment is not solved still in this case we see reliance on the audio but and consideration of variability and about token classification we find soon X-men Robert for two sets of tech binary and multi-class and we see strong influence of variability on the choice of classifier but we see that this approach still cannot cope with lexical and syntactic features next we decide to have experiments with the poshsky data in the rule base it shows results higher than on the podnadvinsky data which may indicate only more consistent implementation of the features in speech of informants and about entire audio classification it is important that the implementation is impossible without training only on the target data only on the the podnadvinsky data and it is necessary to train on target dialect too and last token classification we see that models that X very few dialect features without fine tuning on the target data but it rather good results for after fine tuning on this data and we create pipeline to work with the best models and audio recording in web format is accepted as input data we convert audio to a single channel format then we divide the recording based of areas of silence fragments of audio bevel special decibel threshold and we get the descriptions with the best model to then we also have attack for each token and we generate finally two formats text read and ef to work with chat and event programs and on this slide you can see how it looks like in the event program and conclusion if the goal is to get the model to recognize one selected dialect fine tuning a model that have already seen a close dialect will give a better result then fine tuning a model trained only in standard Russian language and you can see it on the case of opportunity data that's phonetic morphological syntactic and lexical differences between close dialects do not compare the quality of recognition and the last data to fine tune should be the target dialect for which the model is being trained and not some close one otherwise the quality will be lower and we see it in Odvinsky data and new hypothesis has been put forward for further research to create a universal dialect speech recognition model and it is necessary to fine tune the model on the entire sample of dialects at the same time thank you, that's all any questions? I have a question I'm not familiar with scope dialects you know how do you summarize the difference in realistic points of view because I don't know exactly what's the dignity factor yeah, they had a lot of similarities but differences too and they had different realizations of phoneme they had morphological differences for example in Opocitsky dialect they have a final consonant in the sort present form of verbs and in the Pododvinsky data in this case but in Opocitsky data they don't have last consonant at all and we have a list of features and in most of them they have a little differences I guess that we still going to know it for a while yeah could it be that you can artificially generate one dialect from another in those rules in order to increase data in training it's a good question I don't know exactly but I think it is possible but nobody so maybe it's a bit generic question but so this linguistic expeditions could they be somehow modernized now with let's say modern technologies mobile phones and LLMs or something like this let's say you just install certain application and ask some people to interact with some dialect agent and then the data is collected more or less distributed way in a very cheap way could this be do people actually start to think about these kind of practices or there is certain inherent limitation for linguistic data collection in this way now it is not popular at all to use modern instruments and expeditions I was in expeditions expeditions this year and we still go here with micro we write we record informants speakers and then manually annotate their speech but this my work I believe that I have done a step to modernize the process of this and I believe that we will go next year to the expedition and use this tool thank you thank you for the talk quite condensed it's good that we found new hypothesis my question is about this proposal for universal dialect speech recognition model how this should operate what is your vision of this model and what is the purpose of this model should it predict like a multi-label given the speech it should predict like 1 out of 100 or maybe 10 I don't know or something different and why people need this model I think that now we have dictionaries and other sources where dialect features compare with each other in different dialects and I believe that we can find more if we try to optimize this process and also I should say about that some features died and we should see what is still in the dialects we can see okay but is this task can be solved just building some atlas or some like reference of all the dialects why do we need machine learning or whatever all resources was collected 50 70 years ago and it should be modernized and I think that it is one of possible way to do it also questions yes questions regarding the practical application of this interesting instrument first thank you for the talk and you have already said that it hasn't yet been used for like in the expeditions but it could be interesting like first if it's not 100% that if it's not 100% quality maybe it can first be used like to guide the scientists and then they could correct the mistakes and second it could be interesting to look at if the model and the scientists make mistakes in different or in the same places so maybe the model could help them in the specific places where they are unsure yes of course the first step it should be corrected by the experts and then I think that model can reach better result if we train the entire sample of dialects and it is too parallel process for me as I can see and about the second can you repeat yes it could be interesting to look at if the model and scientists make mistakes in the same places or not I analyze only models errors but it is interesting to see notators errors and compare them so I think that it will be good enough to do it in future thank you for the talk I'm curious what are the most notable differences between Pskov dialects and the standard Russian they have unique phonetics for example I talked about first press syllable they have for example then they had unique syntactic structure they have verbs forms that we don't have in Russian standard language and also they have in our mind disagreement in auxiliary verbs and verb the main so there are a lot of differences we can see also in constructions and homophology got it thank you and do you understand correctly that this dialect is influenced by the Belarusian language we have some cultures about it and of course the main influence is standard Russian because of TV people that were born recently but in some cases they have influence of Belarus got it thank you thank you thank you hello everyone thank you for coming today we are talking about compression of large language model based on transformer architectures and compression is provided by matrix or tensor decompositions the field of natural language processing has made significant progress with the development of large language model so we have a lot of factors transformer models share a common challenge of expanding scale presenting an obstacle presenting an obstacle to model employment and training especially for small research groups so in our work we decided to reduce the size of large language models for example the initial layer using tensor and matrix decomposition so we have Bert and Bart and this is model based on transformer architecture and most of parameters several work shows that many parameters into this model are redundant so we decided to take several layers select several layers and compress it we see how many parameters contained in the different layers inside the model every transformer architecture consist of MLP blocks, embedding blocks and attention block and how you can see into this table the concerned Bert and Bart and this is out of scope of this presentation but this is true for GPT-22 for the quarter based model too the most number of parameters are in the MLP block so we decided to take MLP block layers inside this MLP block consist of two fully collected layers and apply several decomposition to it we decided to take one matrix decomposition is a singular value decomposition matrix decomposition with facial information one tensor based decomposition tensor train matrix decomposition and tensor train matrix decomposition with facial information so we made this variant of the composition and replaced fully connected layers inside the proper architecture with the corresponding representation as a baseline we take full model which was obtained by downcasted the full model by PyTorch FP16 in PyTorch FP16 we started to represent our weight not into the floating point of with two precision but with floating with 16 precision and block pruning to do block pruning we select several MLP layers or several head into the attention block prune it, throw it over and obtain a model with the smaller size so how we can apply singular value decomposition into our fully connected layer has a weight matrix weight with the size dimension input multiply to dimension output and we can represent this matrix weight with in the singular value form with two factor matrices U we transposed and with the diagonal matrix sigma which contains of singular values to make shorted version compressed version of it we can select the most significant singular value into the sigma first throw into the U matrix and the column into the V transpose matrix and we obtain the compressed version of our decomposition so having initial linear weight double V we can now we can obtain two U weight double V2 with this multiplication of truncated version of factor U, multiply to square root to the matrix sigma and double V1 with this multiplication of square root of matrix sigma to the truncated version of factor matrix V following and into the denominator we have multiplication of input and output dimension of the initial matrix and as a denominator we have some of the multiplication of weight of the size of this and this weight matrices to understand how we can deal with tensor compression we should go into some tensor notation so tensor is a multidimensional array it's a big cube which have n axis where n is a number of dimension and this big cube this big object can suffer from a can suffer of having a lot of parameters so we can obtain some tensor compression technique to represent this tensor into the more more compressed way in other word we represent the tensor multidimensional array as a set of factor object which usually have less dimensions than the initial tensor and because of this object have less dimension we have a compression comparing to the initial tensor so some notation we want to know by this letter tensor is defined by this letter with where n is a number of dimension into the tensor the tensor in try is defined one element inside the tensor this is the regular definition for the core tensor in some decomposition so when we decompose some tensor into the set of object with low dimensions we this object usually represented by this letter this is matrix and vector and this is this is the definition for modern monetization or unfolding operation of a tensor what is unfolding operation when we make unfolding we create a matrix based on our tensor in other word we have tensor with n dimension in this slide we have a tensor with dimension number 3 and we select one dimension and scratch our tensor alongside the selected dimension like harmonic and we have matrix and this matrices depend on the initial tensor and on the axis alongside we scratch it so if tensor has 3 dimension we have 3 different matrices by unfolding this tensor by every dimension when we want to represent tensor in more compressed way first step is to define the way in which we will represent our tensor and the next step will be to fill this set of containers by the digit by the value in the proper way so we decided to represent our matrix in tensor train matrix format with an extension of tensor train format what is tensor train format this is when we represent our initial tensor by the set of core tensor every core tensor has dimension no more than 3 so out core tensor has dimension 2 but in the core tensor has dimension 3 so the number of core tensor the number of tensor into this sequence is equal to the number of dimension to the tensor so here we have a tensor with dimension equal to 3 and we have 3 core tensors to calculate to count the entry inside the corresponding tensor we should do the same for example we want to calculate the entry under the indexes 2, 3, 1 we should select the second slice here from the second core the third slice here for the first core the second slice from the first core the third slice from the second core and the first slice from the last core multiply it and after multiplication we obtain object with size 1 with number of dimensions 1 we obtain a point the proper point so this is the formula which we use to calculate every entry of the tensor and we also should say that every tensor has no more than 3 dimension the outer dimension is rank rank is a dimension on the right side the neighbor tensor will be multiplying so rank 2 here and rank 2 here is equal the first rank into the outer tensor is equal to 1 and the rank is usually different so we obtain tensor train decomposition with different ranks but for simplicity of formula for simplicity of calculation all rank are usually set equal to each other and equal to R we select our form in which case we will represent our tensor and now we should understand which digit should be in our core tensor how to do it there are several algorithms tensor train compression and one of the most famous is TTSVD which is provided by Ivan Asseliaditz how it works we have initial tensor and we should do initial tensor with d axis and we should do d steps on every step we unfold our tensor over the selected axis this is unfolding then we apply to the obtained matrix SVD obtaining u, vt and sigma truncated to the corresponding rank this rank should be the rank of tensor train decomposition we obtain the corresponding core tensor by reshaping factor matrix u and the rest of our factors we multiply it and this multiplication go to the next step so on the next step we unfold this tensor instead of the initial so we do it the times which equal to the number of axes into the tensor and obtain the number of cores which is equal to the number of axes into the tensor unfortunately we cannot apply it to the neural network architecture and for neural network we should use the expansion of the tensor train format the train matrix format tensor train matrix format is very similar to tensor train format instead of two point if we are talking about tensor train representation we usually talking about tensor over the point if we are talking about tensor train matrix representation we are usually talking about tensor over matrix so we have no point with the dimensionality 1 but matrix with dimensionality 2 and now our index y turns into the index tuple so now we have no core with dimension maximum 3 but we have and cores with maximum dimension equal to 4 and our formula of calculating entry turn out to be this one so we have two index instead of one and we have core tensor with the biggest dimensions and into the tensor train how we we decided which form we will now we have matrix with size dimension input and dimension output then we factorize dimension into the factors our factors will be the initial shape of our cores and then we reshaped this matrix into the two n dimensional object two n dimensional tensor we should permute axis of these objects so the required axis from the tuple of the indexes are adjacent and when we obtain tensor C by reshaping we apply tensor train SVD algorithm on this the compression rate is this so into the denominator we have multiplying of all factors which in real life is a multiplication of dimension input and dimension output and here we have some of multiplication of dimension of every core tensor so returning to our Bert and Bart model we decided to see its behavior on three compression rates and for every compression rate we selected the proper rank for SVD for transcated SVD decomposition and for tensor train matrix decomposition so if we want to update for example 69 million into the Bert we should apply SVD with this rank and TTM with this rank SVD is quite simple we have two linear layers a sequential linear layer instead of one and in TTM we decided to represent our matrix as a set of four cores with these shapes so into the TTM algorithm we can varize the number of cores too so we can represent our matrix as three cores as four cores or as five if you want so with these shapes where our rank we take from this table and we choose jk that j multiply k are approximately equal among all tensors inside the sequence of core tensors okay we were working with Bertrain Transformer based language models and when we applied decomposition of course our quality has dropped significantly so we decided to align task objective which we used to obtain some decomposition of the given matrices with task objective which we have on our work with our model is working to do this we decided to inject fission information into our single decomposition algorithm fission information is the way of measuring the amount of information that an observable random variable x carries about an unknown parameter theta of a distribution that models x in other words we have a set of data set of number of object and on the certain object model gave some prediction so we can calculate loss of this prediction and in this loss explicitly contains information about some parameters inside the model so fission information is defined as here a partial derivative over the probability which over the probability of the output which model gives on the certain object into the data set with respect to w w in this formula is the weight inside our layer and we can approximate this in this way so we calculate a loss on every object take a derivative and calculate the mathematical expectation so we obtained i v i v is a matrix of fission information which has the similar size as our initial matrix v we the matrix of fully connected layers so we multiply this matrix by matrix of fission formation and obtain a single variable decomposition on this multiplication and we should also multiply u-factor to the inverted matrix of fission information in this way we inject information of the output of the model objective into the single variable decomposition algorithm ok, what we should do with tensor-train matrix algorithm it's very simple, as you know tensor-train matrix algorithm is based on tensor-train algorithm and tensor-train decomposition algorithm is effect a set of several SVD so if we have fission matrix i double v we do with this matrix the same operation as we do with matrix v to obtain our tensor and then we unfold tensor based on matrix v and unfold tensor based on matrix i v and we have two matrices and do with two matrices the same things that we do here so into the tensor-train SVD decomposition algorithm set of SVD steps and into the SVD steps we do the same thing that we do here and by this way we inject our fission information into the tensor-train matrix approaches ok we define for decomposition technique that we have seen before and then we started to evaluate it the first evaluation point was built we evaluate our method for the natural language understanding task, global benchmark global benchmark is a benchmark which consists of language assessment, sentiment analysis paraphrasing, semantic similarity, natural language inference tasks we firstly fine tune our model over every task from benchmark for one epoch and then we compressed fully connected layer using one of four techniques and then we fine tune our model again during one epoch and we obtain this score and the score can show that on the big compression rate this part of table the best performance comes from TTM and fissure weighted approaches but on the medium compression rate and lower compression rate the best performance comes from SVD and fissure weighted SVD approaches and also when we apply fission information into one of the SVD or TTM algorithm we usually have increasing in our performance increasing in model accuracy or some other scores the next point was sequence to sequence was a variation of sequence to sequence model BART and the first task for BART was paraphrasing we made paraphrasing of paradox data set in this data set we have a pair of sentences the first sentence is looks like sub-retirations and the sentence we want to obtain should look more polite than the initial phrase on this data set four metrics are works the first is style transfer accuracy how accurate we fit into the polite mode similarity how similar the meaning of this and these sentences and fluency of generated text and the last metric is joint score this is a variation of these three metrics on this data set the best score usually comes on every compression rates comes from fissure weight SVD and fissure weight SVD gives the great boost gives a great increasing into the performance according to other according to other approaches the last evaluation point is also sequence-to-sequence model BART and we try to train BART to provide a summarization we have a data set X sum which consists of several hundred thousand of BBC articles and a single sentence which provides summary of these articles and on this data set so the example this data set imitates the result on the glue the best score on the high compression rate is provided by feed-forward TTM so added fissure information into the our decomposition algorithm also provides some boosting scores on high compression rate the best scores come from fissure weight TTM on the medium and low compression rate the best score comes from fissure weight SVD so there is a graphic for a different task for Glubinsh Mark for BART and on this graphic you can see the main the main tendency of the whole work red is a SVD blue is a fissure weighted SVD green is TTM yellow is fissure weighted TTM so usually the best score is for fissure weighted SVD but on the several tasks on the high compression rate which is described by the right part of this graphic of every graphic the best score goes to the TTM or fissure weighted TTM so this is the rest of our glue tasks and this is the average work of the whole glue so as a result we take four different techniques for compression of fully connected layers in BART and BART models a different compression level and different technique can give better or worse result usually for BART on Glubinsh Mark and BART for XSUM on the high compression rate TTM fissure weighted TTM provides the best score for the variants of compression fissure weighted SVD provides the best score and the alignment of the task and the decomposition objective by injected fissure weighted information inside to the decomposition algorithm can significantly improve the performance of the compressed model so thank you for your attention Thank you very much Thank you I think that distillation is quite good technique distillation is good when you train your distill model towards the desired task for example you could say ok I want to obtain distill version of BART I train distill version of BART to the task of the initial BART to the task of natural language understanding and on natural language understanding distillation can provide good score but when you train your model towards the desired task on the other type of task distillation can provide not very good score and our method gives approximately the same result over all set of tasks yes and TTM too I don't think so you mean the compressed different layer with different drugs and with different shapes it can make sense and we made some research which is out of scope of this presentation which can see that layers inside the TTM and BART into the BART or GPT can be less compressible according to 3D and TTM and can be worse compressible it can be seen by single value spectrum and of course it makes sense it can give some boost but it's quite difficult and we decided to set one rank to the whole model I have a question we have a large language models and the popular method of compressing them is quantization and there is no studies but there is an evidence about that larger models are easier to compress and so have you conducted some ablations about the scales I saw about BART conducted some experiments with BART large maybe it was better for compression I mean the model with large number of parameter can become less better yes no I haven't I haven't seen such paper which provides such experiment I only have seen a paper from open no sorry from META about about binding between the number of parameter into the model and number of talking into the data set on which you train your model with their different scores with the different metrics and people proved that this this function is logarithmic but unfortunately I haven't read any paper which can directly prove that we can compress large language model is better compressible that less large language models yeah and follow up question about all this decomposition there are some related work much older about SVD decomposition and they show that after decomposing the model if you run about one epoch of pre-training it will perform better have you done something similar in your ablations if we train for yes yes so it does help yes absolutely thank you it's very depend on the size of task we selected so people usually evaluate large model not very big data set and when we are working with not very big data set one epoch is really enough to to set model to some good point another question first of all thank you for your talk and I'm also curious about compared to quantization because like in the field of large language models nowadays quantization is very popular and I know that for example 16 model strain in 16 bit precision they are like can be compressed for example to 4 bits almost without losing much quality so basically we will lose something like 1-2% of perplexity for 25% so I'm curious how your approach is compared to these advancements you asked about comparison of quantization and my approach or can we apply our approach with quantization comparison so which one is better I suppose we have information in slides because we didn't compare with 4 bit quantization but for 60 floating point and we can see that for exam then our approach provides the best score then FP evaluation this one this this role into the paraphrasing paraphrasing something change and FP16 provides the best score than every of our approaches I got it I think one issue here is that here we see like naive so basically all the bits were in float 32 and were naively converted to float 16 but like this new approaches they are kind of smart for example they take out liars and they quantize some percent of outliers differently and so on and they provide like I think it may be interesting for you to compare with them because they provide really good compression rates without any significant harms to quality we have most sophisticated ways of quantization and I think it's really interesting yes to compare with this yeah thank you this is a really interesting alternative approach hello dear colleagues today I'd like to present to you the results of our research which was devoted to the development of the geothermal recommendation system the title of our paper is on the slide the authors of the paper are me, Dmitry Hrnoklev and my colleague Pavel Prostovsky let us begin with the main idea behind our study as most of us probably know idioms are quite an essential part of many languages and native speakers usually tend to use them in certain circumstances because idioms enhance the fluency of speech and expressiveness of speech however no native speakers may sometimes struggle to find an appropriate idiom for some context also I'd like to mention here that nowadays become more and more popular, automated writing assistants writing assistants some of which are powered already by the way like D-bell and Grammarly are able not only to correct mistakes grammar mistakes or spelling mistakes but also they are able to improve style and suggest continuations for the entered text and so here we come to the motivation of our research we aimed to suggest system which is able to recommend an idiom for a given context on the upper image you can see for writing assistant and below there is a kind of simplified scheme of what desired automated system should look like and how it should function for the purpose of our study we obviously needed data and by this I mean a set of idiomatic expressions and contexts which contain these idioms based on the literature review we have chosen EPI dataset which is the basic corpora in our study we have chosen it because of the several reason firstly it is the second largest freely available dataset another reason is that it contains definitions for the idioms that are presented in this corpora table added up of the slide contains several examples from this dataset however this dataset required some sort of preprocessing which is described in detail in our paper after it we have split the result in dataset into train test subsets and in the 88 to 20 ratio with stratification just to make sure that all the unique idioms are represented in both sets also I'd like to mention here that the test set is fixed for all definitions which I'm going to describe further to have a right to compare different configurations during the experimental part of our research we finetuned one of the considered model and in order to make it more robust we required additional data to get more context for our idioms we have used the Guardian API which provides handy interface to parse articles published in the Guardian newspaper this allowed us to obtain almost 25,000 additional sentences and therefore we have increased our initial corpora by more than two and a half times the graph on the right of the slide presents two distributions of the number of different contexts per idiom before and after the parsing process on this slide you can see a scheme which represents our proposed approach our proposed approach is based mainly on the semantic similarity search ideology we obtain embeddings for input sentence and all of the collection sentences input sentences are also called queries in terms of semantic similarity task and the collection sentences are called documents then we obtain embeddings using some model for both of these input sentences and collection instances and then we rank the collection items based on cosine similarity to the query and take the corresponding idioms as recommendations the main metric in our research was mean reciprocal rank just in case if someone is unfamiliar with it here is a formula now before we move further I'd like to mention here just one more point related to the test set it's obvious that when our approach is used on inference we receive sentences without idioms so we remove original idioms from test sentences it's illustrated on this scheme now let's return back to this to our main scheme we can conclude from the scheme that our algorithm has only two key parameters which can be varied the collection on which the semantic search is performed and the model which is used to obtain the embeddings so using different collections and models we obtain various configurations of our main approach let's discuss different collections first in our research we consider four different collections the first is called idioms and it consists only of the unique idioms from the initial data set themselves then we have idioms plus test collection which consists from idioms with concatenated corresponding definitions for these idiomatic expressions then we have sentences collection which consists of the sentences from the train set and finally sentences plus plus collection which consists from the instances from the sentences collection so it's basically just a sentences collection which was enlarged or extended with examples from the Guardian API in the tables on the slide you can see some examples from the collections that I've mentioned now let's switch to the models in our study we employ pre-trained on the Google news dataset word-to-vec model from the Consim library as a baseline model just to establish some kind of power against which we could compare certain configurations in the case of the word-to-vec model to obtain sentence embeddings we've averaged embeddings of the words in the sentence but the main model in our research was sentence birth model since it is considered a state-of-the-art model in the task of semantic similarity search the scheme on the right site of the slide presents sentence birth architecture at inference just as a reminder also in our research we use several sentence birth models straight out of the box from the sentence transformer framework which includes Minilem model sentence birth based on Minilem model sentence birth based on Distiller-Bert and MPNet besides we also finetuned the sentence birth model based on Distiller-Bert which is noted as Distiller-Berta plus to obviously achieve better results now let's talk about the finetuning process of the Distiller-Bert model as I've said earlier we have parsed additional data so we join our initial train from the ebi data set with contexts that we have collected from the garden then we split this new set into new train set and validation set in the ratio of 92 to 10 again with certification then we create so called positive and negative pairs to create a positive pair for a sentence from new train set we merged a sentence with another random sentence from the new train set with the same idiom and then we have removed the idiom from the first sentence the process for creation of negative pairs was identical except that we matched two sentences with different idioms this process is illustrated in the table at the bottom of the slide on the right side of the slide you can see hyperparameters we have used for finetuning and the graph which illustrates the dynamics of the accuracy on train cross 5 epochs so as we can observe at the fifth epoch accuracy reaches the plateau so we have stopped training now let's take a look at the final results this table right here contains MRR scores for all of our configurations as we can see in a green cell finetuned Distilla-Roberta model achieved the highest the highest result overall on the idioms collection as I've said earlier we consider word to word model with sentences collection as a baseline configuration therefore we can see an 80% gain compared to the baseline and 46% gain over mpnet at sentences configuration which achieved the highest MRR score before the finetuning process so we can draw conclusion that MRR is higher than 0.5 which means that on average the correct idiom is ranked second on this slide you can see examples of simple and difficult idioms for our best configuration simple idioms are characterized by a high average reciprocal rank averaged overall corresponding sentences from the test set while average reciprocal rank for difficult idioms is close to 0 as a possible explanation for this for this variations in performance I can mention that first of all some idioms might be used in a wider context or in a more complicated context or perhaps another reason is it could be related to our evaluation protocol because we assume that there is only one suitable idiom for each sentence which might be overly strict however these hypothesis are under-researched and we plan to examine this phenomena in the future so the main results of our study we'd like to highlight that first of all we automatically expanded epi and dataset by more than two and a half times and therefore we have created basically a new dataset for the task of idiom recommendation for the English language secondly we present a fine-tuned model particularly for the task of idiom recommendation thirdly we present a novel approach which is based on the semantic similarity search and implementation of centers bird and also we have examined the suitability of several neural models including word to egg and centers bird for the task of idiom recommendation as for the future plans in the foreseeable future we plan to further expand the dataset because we have a hypothesis that showing the model even more various contexts on the training stage might result in even higher performance and secondly we would like to add filters to prevent some kind of inappropriate recommendations thirdly we would like to analyze the impact of the context length on the performance of our approach and experiment with contexts longer than one sentence and finally we plan to use some kind of tool to filter sentences which contain idioms used in the literal sense because in the initial EPI dataset some idiomatic expressions were used almost exclusively in the literal sense on this slide you can see a QR code which leads to the code models and extended dataset which are all freely available we invite all of you to come and see it that's it thank you for your attention I'm ready to any questions maybe we have time for one quick question because we're slightly out of time good but I have many but yeah I will try to do it quite fast so my question is about the test set because there are multiple ways to split the test set sometimes you may choose the sentences that were in the train with the same phrase but it's not seen in the test set so like can you please elaborate a little bit how the test set was split created and thank you for your talk it was just a random split we didn't have no duplicates because we have removed it if I heard your question right so it was just a random split we just have some contexts containing idioms in the train set and some contexts without idioms in the test set which we want to find the correct idiom for okay thank you and one more quick question like do you have I think that the scores that are higher for the this fine tuning step is that because the structure might be quite evident for the model because some phrases you cannot like food for thought maybe use only with some preposition that you don't hide and some phrases that use a verb that could not evidently according to the grammar cannot be used there so like probably like have you thought about that and maybe it's going to be for future work or maybe there are some examples that they were phrases that were correct but not identified by the test set thanks as for the first part of the question yes it's up to the further research we have thought about it but we haven't got enough time to check all the hypothesis some of them are described in our paper but it's too it's not it's hard to put them in a few words so I can just invite you to read our paper and as for the second part I have mentioned that our evaluation protocol isn't quite as logical because we don't consider the fact that some contexts may contain more than one appropriate idiom so there is not only one correct answer so it means only the fact that we have kind of lower estimation for our approach it might be even better okay thank you let's thank the speaker again and the session is over see you all tomorrow


and welcome to our channel. My name is Andrea with Food and Mentory Adventures in Food. And today I'm here with an unboxing, but it's a little bit different than our typical unboxing. So normally when Howard and I do unboxings or taste tests, we purchase those food items with our own money. But this time a company called Public Goods reached out to us and asked if we wanted to try some of their grocery items. And so check out what we found out on their website. Public Goods is a one-stop shop for everyday sustainable essentials. You can shop for household items, pet essentials, grocery items, and more. So here are all the different categories that they have in the grocery items. And here are some of their offerings. You can see that they have macaroni and cheese. They offer flour, sugar, coconut oil. They have gluten free options and lots more. So they allowed us to choose five grocery items. And the first one that we chose was the shells and cheddar cheese. Howard and I love macaroni and cheese. And I like the fact that this one comes with a creamy cheese sauce. It's not the powdered stuff. And so let me just kind of show you a little bit what's on the back there so I can get it to focus. So we are excited to try this out. Another thing that Howard and I love, if you've been watching our channel for a while, is Asian food. So this is a five pack of ramen noodles. And we selected the spicy sesame oil. And I think they had two other flavors that you could choose from as well. And these were actually made in Taiwan. So we're both super excited. You see that? To try these noodles out. And then Howard was very excited about this hot sauce. They had a lot of different hot sauces to choose from. So he was excited off this one. Because he loves habanero. And this is garlic habanero. And then we also are trying out their tomato sauce. And this I think is a product of Italy, if I remember. Yep, product of Italy. It's a pretty big jar, 23 ounces. And it is a tomato basil sauce. So we are excited to try that out. Because we love pasta and things like that. And then the last thing that we selected were these cookies. And these are chocolate chip cookies. And I'm actually going to try these out right now. So let me just get these open. So these are actually made in the USA. And you can see the pretty short list of ingredients right here, which is pretty cool. I opened up the bag. And it is filled to the brim. And these are my favorite kind of cookies. Because I love a crunchy cookie. So let me try. Very good. Very buttery. I think these are milk chocolate chips too, which I like. I know a lot of people like dark chocolate chips, but I do like a nice milk chocolate chip. But anyway, that's it. I just wanted to show you all what they sent us. I will leave a discount code in the description box if you guys want to check out their website and see the awesome products that they offer. I am not getting any kickbacks at all if you guys decide to order. But it's definitely worth checking out. So I hope you guys enjoyed this video. If you did, please make sure to give us a thumbs up. And if you have not subscribed to our channel, we certainly love to have you. Thanks so much for watching and we'll see you all next time.