Good morning, dear colleagues. I suggest that we slowly start. We are very happy to see that many people relatively early in the morning, after I guess many people had a great evening yesterday. So it is our great pleasure to have our third keynote speaker, Professor Samuel Horvat. Samuel is currently working in the University of Artificial Intelligence in Abu Dhabi. Before that, he did his PhD in Saudi Arabia in Kaust. Samuel is a very well-known expert on distributed and federated learning. He will be talking exactly on some aspects of this topic today. Samuel, go ahead. Thank you, Maxime, for a very generous introduction and thanks to all the organizers for inviting me. What I want to talk about today is federated learning. Let me just briefly discuss the outline of my talk. First, what I'll try to do, I'll try to give you some kind of gentle introduction to what we mean by federated learning and how it developed. I'm going to discuss several practical and research challenges that need to be addressed in order to deploy federated learning in real-world scenarios. Then I'm going to talk a bit about our recent work that we did on decomposable models that are targeted to face some of the challenges of federated learning. Let me start with the motivation of why we do federated learning. If we look at traditional machine learning, originally what we did, you would have a single machine, you would just put all your data in and try to train your machine learning model, but as we progressed, what we see that essentially this model gets bigger and they require more and more data. Therefore, we moved from local solutions to cloud-based solutions. Now the thing with that is like with this ever-increasing data collection, what we have to consider is that a majority of those data comes from clients and that data collection might have some negative privacy implications of these data collections. On top of that, many countries do have different privacy initiatives such as GDPR in the European Union or there is CCPA in the US that essentially forbids direct data collections. With this in mind, essentially, if we want to further progress much in learning by collecting ever more data, that might not be a standard solution, might not be a feasible solution because not respecting those privacy initiatives, we can almost completely lose the access to the data. Now this is where fair-to-learning comes to save us by bringing training actually to the edge, so to those clients that own data and the main premise of fair-to-learning is following. We do assume that we have some orchestrator who orchestrates the whole training process and what it asks those clients to do, it asks them to compute some kind of updates to the model when we do training and those updates are supposed to be very focused and they are intended for immediate aggregation, so essentially this orchestrator or you can think about as a central server only sees the aggregated information in order to prevent any data leakage. So therefore, fair-to-learning is kind of its basic definition what it gives us, it gives us at least a hope to train on this large decentralized data sets. On top of that, there is an increasing demand to have something what's called data locality paradigm, meaning that the data should be processed where they were actually collected and on top of that there are several recent studies that show that when you care about the carbon footprint of your model then if you design your fair-to-learning algorithm well, then you actually have a lower carbon footprint when compared to standard centralized learning. So some of the kind of success stories of fair-to-learning so where it's already commercially applied, this is in a big industry players, for instance Apple have it in their hay series or quick type, Google uses kind of the same thing for Hay Google and G-Board, then where we see this kind of fair-to-learning as a next game-changer, so this is pretty much all applications that historically wouldn't have much of a data and the reason for that would be very strong privacy constraints, this might be, let's say, smart health applications, there are already several startups working on that such as DoKi or Okin and also a lot in fintech applications where essentially privacy is a key and there are several banks already working on that, how to do kind of decentralized fraud detection, for instance WeBank. And I mean the thing is the applications of fair-to-learning, you can think about that as essentially any machine learning task that you would like to do but you have to collect a lot of private data. The good thing about machine learning and why it's kind of getting more and more widespread is several open source framework for fair-to-learning, among the most popular you have a flower, FedML also Meta has its own fair-to-learning simulator same for Microsoft and also quite popular is the NVIDIA Flare which is the SDK for fair-to-learning. I'll also do a bit of an advertisement, so when you're looking for the resource to get to know more about the recent research about fair-to-learning, so we've been running for I think roughly three years, so ever since COVID we are running the online seminar where we have more than 100 talks about different aspects of fair-to-learning there all on YouTube. So if you are interested just Google Flow seminar, register and you will have access to all the talks that we have. And also some of the great resources are some recent review articles or there is even a recent book on fair-to-learning as well. All right, so when we talk about fair-to-learning there are two main settings that we usually consider. So first is something called a cross-site of fair-to-learning and what we have in cross-site of fair-to-learning we have different organizations. You can think about them as a kind of a big organizations that want to collaborate. For instance, the example that I have here are hospitals that have patient records. They would like to train some smart health assistants based on all of these records but we have to realize that the data are private so what you kind of care about here the most is the privacy of the data. So and here we usually assume that the kind of a number of the institutions that try to collaborate is relatively small and the main thing that we would care about in this setup is privacy. So the setting that I will consider more in this talk is cross-device fair-to-learning. So in cross-device fair-to-learning what we have so all those clients that we have are devices you can think about them as mobile devices or different IOT devices and they want to collaborate in order to solve some machine learning problem. You can think about that as a recommendation problem where the whole process is orchestrated by a central server. This one you can think about as a provider and what's their goal? What's their goal is to train this model in this fair-to-learning way where we only communicate the very focus update that are intended for immediate aggregation and also one other thing that we have to respect here that this is all trained within those devices and also the model that we eventually going to obtain is going to be deployed for those devices and one of the kind of main challenge here is essentially the different heterogeneity that comes with the client and also the fact that the number of clients you can think about that as a number of phones that might be in millions or hundreds of millions. All right and that brings me to this kind of second part of the talk where I would like to discuss challenges and challenges that essentially makes or challenges that if we can address those we would be able to make fair-to-learning practical. So the first one and one of the most prevalent one is the communication bottleneck. So for those of you so the example that I list here is for distributed training. So for those of you who are familiar with this we know that essentially just by adding more and more GPUs that doesn't necessarily leads to this perfect linear scaling and the reason for that is at some point what you're going to hit is the fact that communication is actually much slower than computation. So for instance I have in this example right here what you have is essentially if you run let's say deep light which is a very communication heavy model on AP100s with still relatively fast network then what you're going to end up with is that essentially more than 90% of your training time going to be spent on communication and in many applications this would be the major limiting factor. Now if we move to fair-to-learning now here this issue is even more prevalent because now we do not have those machines, those clients being in a single data center. The issue is that they are all connected through some wireless links or some other end-user internet connections and this would be even slower than the data center links. Also on top of that you operate in a system where you have actually a lot and a lot of clients so even the kind of the capacity to aggregate those updates becomes the bottleneck and especially what we see in the real world application is that the model download is still kind of a slow so when the clients download the model from the master or from the orchestrator but then the uploading is the key limitation of the system. Thankfully there are several remedies that one can employ so you can think about the communication compression which means that essentially those updates from the client before they are sent back for the aggregation they're actually compressed and there is a nice line of research linking this that if you design your compression very well you can essentially add more privacy into the systems then another possibilities are that essentially you do not communicate very often what you can do you can try to define the local problem that is some kind of harder than just giving the local update by local update you can think about just computing the gradient with respect to your local data but you're going to do kind of more local work such that you reduce the amount of communication or what you can try to do in some smart way you can try to limit the number of devices that will communicate and what that smart way might be you can think about like how to kind of figure out which clients might have updates that are more important than others but still respecting the privacy of the clients now another issue and that's something that we're going to look at later more in detail is the system heterogeneity so the thing with system heterogeneity is like when you train in this ferretive network a lot of the clients might be either unreliable or very heterogeneous so and this vast heterogeneity is also one of the key challenges when you want to deploy and train these models in kind of in this world so the remedies that you can hear incorporate is try to kind of devise the algorithms that would be stragglers resilient meaning that like if you have some clients that kind of fall behind the training schedule you can still allow them to update the global model later also very popular here is the asynchronous updates here the challenge with the asynchronous updates that usually kind of once you have asynchronous updates it's much harder to guarantee privacy or another thing is like when you have the devices that are not able to compute and even like a store or work with the model then you can just simply drop them another challenge that also quite prevalent with ferretive learning and maybe not very standard in standard centralized learning is client availability so when we talk about this cross device ferretive learning what is the standard consideration is that in order not to reduce the user experience you only participate in training when you are connected to FAS network also you are connected to the charger and also with you there is a large number of devices connected because we want to kind of hide your updates such that your privacy doesn't suffer and with this essentially it creates quite non-trivial constraints and with that you have actually a lot of variations that might be undesired meaning that they actually can kind of break your optimization algorithm and this is something that needs to be addressed another issue with generally like with deploying ferretive learning systems onto devices is the fact that like systems are really not very mature so for instance kind of what's already happening for quite a long time is actually on device inference so that's already deployed in many devices but a full kind of ferretive learning loop is still kind of a work in progress but it's we are kind of moving towards there and nowadays there is actually a couple of providers already allow to run like a forward and backward on the on the devices another issue is the issue of limited labels so while we do have like when you think about your mobile phone like you do have a lot of let's say texts a lot of photos that maybe we can train on but the thing with that is there's very few labels and that's for instance one reason if we would go back to kind of this first applications which was next word predictions for instance for G board the thing is there you don't need labels and this is why kind of the first applications of ferretive learning works for next word prediction now so line of work that people looking at is how to incorporate semi-supervised learning while exploiting the data structure within the federated learning how to actually incentivize clients to label data and then okay and then we can talk about personalization so now the first thing about ferretive learning is that we might be actually training on very different kind of within the clients we might have very different subset of data and then the question is whether the kind of a global model which is kind of standard way of doing ferretive learning is the right thing to do so let's say when we look at this application right here which would be next word prediction that's very much depend on the context and of the uses of here that would be not a case so with the social kind of a possible direction for the research and things that people looking at is how to incorporate kind of a meta-learning approaches into ferretive learning there's actually a lot of work on trying to kind of link ferretive learning and meta-learning together and also how to kind of discover kind of interesting structures within our kind of a federated network by looking at the clustering or some kind of a balance between local and global model whereby local you mean you kind of think you can think about that as a model that trained purely on the kind of a distribution of the data that you honestly locally and globally that would be the global distribution and then other kind of popular subset of the ferretive learning as a split learning another kind of a very important issue that maybe was not discussed at kind of the rise of the ferretive learning or when it was originally introduced is privacy guarantees so the thing with this is while we have by construction that the data they actually never leave the device we do not have any formal guarantees that from the updates that we sent even though they are aggregated and there's like a lot of techniques to make them kind of private we do not have or we did not have originally formal guarantees that ferretive learning does not introduce any privacy loss and also when you think about that essentially you cannot have zero privacy loss if you have the learning so and then also there was a couple of works that essentially showed that even when you do ferretive learning in a way that you try to have very focused updates you average or aggregate over many clients still if you are smart then privacy leaks are possible and this is where remedies such as general cryptography comes so that is more like a kind of having a system more safe but when we talk about true privacy of the clients that's where differential privacy pops up and for those of you who are not familiar with differential privacy what differential privacy generally says that kind of the informal definition is that if I train the model and if you use my data or do not use my data the actually output of that model doesn't change much and by how much change that essentially like by the upper bound how much it can change that's the guarantee that you have in respect of differential privacy now another issue that comes with a ferretive learning is that essentially we can think about ferretive learning as a collaboration among many and potentially mutually entrusted clients so it's very easy target for poisoning and you can think about that essentially because everything is open you might have a competition people might be training the similar model so there is incentive to kind of destroy your competitive model so when designing the ferretive learning algorithm that you're going to deploy in a while you have to be aware of that but thankfully so in order to kind of prevent that there are several defense mechanism that can at least remedy this poisoning attacks now the last thing that I want to discuss is about the challenges as this intensive incentives to participate so when we think about like why would I even kind of give you my data to participate in this like ferretive learning training that's something that has to be somehow clearly defined also with respect to label, for instance like your model would improve if I label the data but why would I do that also with a local training there is a cause incurred with like energy privacy and so on and also another like last issue is also whether I do even benefit from the federation now also that's kind of from the point of a client but from the point of this kind of orchestrator we also would like to know like what's your contribution so do you even contribute to federation like by not having your data do I have any benefit and our issues comes with like the ownership of the model so who actually like owns the model is it a community or is it some kind of this global service provider and this actually opens up a very interesting question especially if you work on economics so in general we already have kind of a when we talk about these three edges which is computer science, economics and statistics we have already like a lot of discipline when it comes to when it comes to the intersection of two such as the economics plus computer science would be algorithmic game theory then statistics with computer science is essentially the basics of machine learning, statistics plus econometrics but the thing is when we look in the middle so this is where we kind of see the fairytid or collaborative learning when addressing all of these issues so with this I hope that I at least a bit appreciate you that fairytid learning presents kind of a new realm of unique complex challenges when we want to overcome these we must devise some kind of a system aware efficient optimization techniques and the key area to focus is optimization theory networking and scheduling techniques and with this in a hope that these techniques will help streamline this data processing and improve our performance and therefore enhance the overall effectiveness of fairytid learning models alright so that was kind of that was the first part and now the second part that I would like to share now the second part that I want to focus on is these decomposable models now with this I would like to start with the motivation so our motivation comes from like a very popular linear algebra techniques that are widely used in machine learning so there's principle component analysis and singular value decompositions so why these are very popular because they can give you dimensionality reduction noise reduction you can extract features and also you can significantly improve your computational efficiency and now the question that we start with is what we can make kind of a PCA or SVD version of neural networks so in order to answer that question let me first kind of try to look at the SVD how we could potentially represent that as a neural network so if you look at this decomposition right here then how we can have it as a neural network is we can consider the neural network that has one hidden layer without any activation, without any bias that would essentially represent this mapping now the nice thing that you can know about that is like if we do this kind of a pruning within this hidden layer then what you can get you can get the single body the SVD but the reduced SVD where we only keep the first let's say here two singular vectors and so on and so forth now so this at least kind of gives us the good example that we can represent this matrix factorization within the neural networks now maybe the most interesting part is whether we can actually learn this and the thing is that we can so if we just define kind of general matrix decomposition we can define it as a following optimization problem now if you want to retransform it into SVD then what you can, I mean SVD would essentially solve this problem individually for each kind of each rank K so here this one means the first K columns of matrix U now the nice thing with that you can even put it all together so you can have it as a summation across all the ranks I mean summation, the nice thing about summation summation you can represent with expectation and once you see this and you're already familiar with what we do whenever you have many terms in machine learning what you can do you can just simply apply SVD and that leads to our construction of what we call the order dropout so essentially what we show if you define your problem in this way you apply SVD but SVD not in the data points you essentially apply SGD to the models then you can actually learn SVD with a standard kind of machine learning training loop and also the nice thing about that if you're interested in optimization you can still see this problem as an overparameterized problem although even it's kind of a nut because the gradient at the optimum even for each sub-problems are zero now how does this look in practice so in practice what we do is like if we were about to learn the let's say like SVD so we would apply this order dropout techniques in the hidden layer and what in each step we do we have some distribution over the width in each step we sample the width and that's the number of neurons that we keep so essentially that's what we call a dropout and not the dropout because it preserves the natural order of the neurons now the nice thing about that is that here you can recover I mean with this construction as I saw you can recover SVD as a special case so for SVD if you have your mapping between the data and the output to be linear and then you sample your data from the uniform ball that essentially you can say that the training loop like the standard training loop with this network with order dropout recover SVD as a special case and that's what I have here so essentially you see that across any of these like sub matrices so here that would be essentially this rank K matrix that it converges to the best K rank approximation of the matrix A within the single loop and it's also within the sampling so we do not need to evaluate with respect to every single matrix but we can just sample one at a time now also you can recover PCA with this in this very kind of a similar scenario where the only difference is that now I'm going to sample uniformly at random from my data set but the thing is the mapping between the features and the labels essentially that's the same thing here and what you can see, so here is the example that we have here is we only have the data to come from like a three dimensional space and you can see that the quickly the network discovers that and essentially provides you this provides you with these three principle components and zero out the rest okay and then in general case that's I'm not going to discuss much about that but essentially if we talk about this general case for this linear network there's something in between what it does is essentially does PCA on the transform data set by the mapping A now how do we generalize that to neural networks that essentially comes or boils down to these two works that I'm going to present so first this title Fair and Accurate Learning under heterogeneous targets where we're going to use the order dropout and this is essentially the work where we introduce this notion of order dropout so first let me acknowledge my collaborators here Steve, Mario Stelios, Elias and Nick just to remind you the problem that we're trying to solve here is we're trying to look at heterogeneous devices so we want to address the issue of having different tiers of devices within our training but we do want to avoid this kind of a standard construction where when you look at freight loop in general that dividedly accepted norm is that the model that you have so all the local models everything that you're going to deploy has to have exactly the same architecture with the global model so in order to achieve that you can drop load type devices or essentially limit the global model size in order to accommodate all your clients now the drawbacks of those approaches that I just discussed is first when you drop load type devices you might have very limited participation and also now freight learning is kind of training on also not just like heterogeneous devices but very heterogeneous data so by dropping load type devices you actually might lose a very important part of your data set and therefore have a lot of bias of your data also by and then by limiting the global model size you might actually degrade performance for high tier devices now what will be our goals and what we want to achieve with this work is we want to have a fairness in participation meaning that every single client can participate and also we want to have very competitive performance meaning that all devices should have as good performance as possible during their local network constraints now how do we achieve that so I mean very kind of naive way is first we're just going to simply drop this assumption that every single client needs to run or store the same global model and what we're going to do is for these lower tier devices we're going to deploy the thinner models I'll show you how we do it later where the width that we're going to deploy will depend dynamically on the local network and network constraints that might be memory, computational capabilities load, battery level or let's say limited bandwidth so and how do we achieve that it's exactly through the order dropout that I just discussed so here so if we have a general network as the one on the left then what we can do is we can apply our order dropout technique in the following way so what we're going to do we define set of relative submodel width so these are numbers between 0 and 1 and we define distribution over those width and what we're going to do in each step in each step we're going to sample from the distribution and based on the width we're going to kind of apply that width limitation into every layer but the input and output layer we're going to keep the structure of the network and change now just to give you kind of a brief motivation or the how this relates to kind of a standard random dropout so for order dropout the motivation is not necessarily only regularization but the motivation is that once we apply this technique we kind of enforce as much knowledge as possible towards the left of the network random dropout the motivation is to prevent codependence of the neurons and also another kind of a difference between the random dropout and the order dropout is that our inference is exact where random dropout has an exact inference where you kind of uh where you just approximate the ensemble with a simple average of weights those are kind of the main differences now just to show you like one example so consider this is our network right here so we have the sampling let's say that sampling sample 0.4 that means we keep 40% of each hidden layer and which here yields two neurons per each hidden layer that we consider and then once we're going to deploy our train then we can accommodate the width of the network based on the tier of the device that we are about to deploy to now how do we apply this into kind of a fairly learning training loop so the construction is following so we have a set of our devices we're going to first have we take the architecture we take a set of the width that we want to train on our model and also the distribution of the width that we want to sample from here example is for instance like let's look at Cypher 10 on ResNet so we might have a file width that we want to train on and you can see that essentially leads to different number of max and parameters per network across different width and what we're going to do for the devices we essentially split them into several tiers and then what we're going to do each tier is assigned with the pmax value which is essentially the maximum width that those devices might work with while respecting the constraints now what we're going to do then for a single communication route or single training step of our training as standard we're just going to select the devices to participate based on the constraints that I discussed then for each device we send the model that corresponds to its pmax of the maximum width that it can work with now it perform local steps but with the order drop out in order to kind of train across all the width that it's kept able to train on just because to get the proper solution to get this kind of a decomposable network as I described before once we have this they obtain the update of the model that's just communicating back and then the server aggregates it's kind of in this non-uniform manner in a sense we just aggregate over the client for the given set of width we aggregate over the client that essentially updated it and then for the inference you just deploy based on the device capabilities now one of the nice things because you already have the model that train like that that's actually decomposable is that even after deployment imagine that now we take this device from the higher tier even if this phase we can then scale the model dynamically during the inference for instance like if that model has current increase or like has decreased battery level or increased load then in order to kind of perform within the constraint we can just simply decrease the size of the model because we train it in this decomposable manner okay so let me show you some of the experiments so here okay so that's strange so here the one on the right I'm going to focus on the cipher only so here what we did essentially we're going to just use the same training setup as for this model right here and what we're going to do we're just going to train it with the order dropout and what are these red squares right here that's a model that's optimized and trained from the scratch so here the red models essentially where you have you have five different trainings while this one the orange one is ours just training by a single loop based on the hyperparameters of the bigger model and what you can see essentially performance is more or less on par and we believe like with the tuning if kind of like a regularization hits in that if you would tune a bit of parameter then you can actually have even better performance than just train models trains from the scratch now another thing is what might be maybe a bit or what somebody might say that essentially like okay do we really learn the meaningful the composition of the model and one way to check it we take our model train with order dropout and we just compare it to the model that I was trained only for a full width and we're going to obtain with like random dropout we're going to obtain the models that have the same number of parameters as our model with order dropout and look at its performance just kind of to double check that it's not just some kind of implicit property of a general training that you discover that essentially like any model of that with your sample would have the same performance and here you essentially see very huge drop even if you take like 40% of the network width and essentially that corresponds pretty much to random guess or we can still do more than 90% apologies that seems that from the neck to windows doesn't like some figures so here the next thing that I want to link here that's a nice property of the order dropout is that you can actually increase the granularity of the width that you train on so when we go from the uniform grid of size 5 to uniform grid of size 10 and essentially you can see that the performance across the intersecting width variable intersects and and then once we actually going to deploy this into further learning where we see the best increase in the performance where the reason is that essentially the only available baseline at the time of this work was to kind of consider that in order to accommodate the clients that cannot run the given model we just when we train the model on that given client we just going to apply random dropout and that's the way how we going to match the constraint of the client but with that you can actually see by trying to train the larger and larger models you actually might see the decrease in the performance while for us you can see the steady increase of a performance meaning that the more compute power or more like parameters you can store the actually the better model you can get and this is also just to double check that the scalability also transfer to the further learning now what I'm going to discuss quickly is this next work that also is based on on the order dropout where this kind of motivation or kind of usage of the order dropout is slightly different so again to acknowledge my collaborators Steve, Shoshang and Hongi so what's the main challenge that we're trying to overcome here is actually how to train large deep models or like how to make kind of training of large small deep models kind of equivalent to train large deep models and if you look at the standard problems with training large deep models particularly those one with billions or millions of parameters there is energy consumption, resource demand and data requirements now so the thing is like why do we actually train large models the reason is that they perform so well the reason for that is this is still kind of active area of research might be due to implicit regularization meaning that the model biases towards the simple solutions and also smoothness in the last cave so it's actually easy to optimize larger models than smaller ones now what would be our goal is whether we can kind of keep the benefit of training larger models where we actually going to train smaller models and then the aim is to design models that can maintain high performance and reducing the size of computational requirements and how we are going to do it we are going to do it through order dropout and we are going to try to discover some kind of low dimensional structures via this efficient decomposition and by low dimensional structures here I mean a low rank of the weight that's based on the prior work that I essentially observed that that's kind of one good notion of low dimensionality alright and how we are going to do it essentially we already see that the dropout is kind of connected to the decompositions so what we are going to do is if we have original mapping and we transform it into factorized mapping we will try to decompose it try to essentially remove the zeros and that's how we get the low rank approximation and how we make this kind of low rank approximation nice and trainable this is the way how we design it with the order dropout so let's have this our original network what we are going to do is within the each layer we are going to input the layer that has an inner dimension of a minimum of these two layers then what we are going to do we are just going to input this as a factorized layer where we are going to deploy order dropout in this factorized layer the order dropout here doesn't have any bias or any activation for training now our sampling is going to be a bit different what we are going to do is we are going to sample one layer at a time so here that might be the first one and we are going to sample a rank so this is essentially in each step the network that we are going to evaluate and the reason for that is to essentially learn an efficient factorization of this layer that we can later prune and that factorization is essentially the one that should account the data for as well now that essentially these are the main component of the method that we propose here called maestro so just to walk you through the algorithm so what we do here so essentially this first part right here is just an order dropout so this is just a sampling and obtaining the prune network then something that I didn't discuss so in order to have essentially low ranks we somehow have to enforce it and one good way to enforce low rank is lasso penalty and because we have a natural ordering we can do group lasso and then the last thing to actually have a good adaptive pruning so to actually prune as we train we have this condition right here so essentially what it tells like once the network discovers that I don't need certain ranks I'm just going to simply drop it now let me quickly walk you through the experiments so essentially what this table tells you that essentially with any other method that does low rank approximation with just under SVD but not the decomposition that essentially tailored to the data and the network we do much better and we can do much sparse networks as well then also what I want to show here essentially it's here the one in the middle what we display here is ranks per layer for a different group lasso penalty and what you can actually see here that it might be not that the network learns only the composition within the layer but also some global decomposition because the thing here is that by increasing the group lasso penalty you always get a subset of the original network another thing just kind of a sanity check when we compare like when we kind of design a pruning technique because again you get a decomposition so even with having a network you can do post-training pruning because you have decomposed network already we can compare it to SVD which is kind of a naive linear approximation when we go for the decomposition so we can see that we do much better there and also another interesting thing is that we actually don't need to kind of look for a good pruning technique what we can actually do we can just look at the other networks found and I'll try to replicate that and it actually works relatively well okay so I'll just have two more slides just to summarize that we discussed two main one of the two key challenges infrared learning one is the efficient training and inference and heterogeneous devices we looked at all the dropout as a technique that would enable the composition of networks and we introduced these two techniques to exploit order dropout that is fjord and maestro and some other interesting application of order dropout that we're looking at right now is the automatic rank selection for Lola some model alignment once you're architectures and also network consolidation so with this let me conclude and thank you for your attention thanks a lot Sam for this amazing talk colleagues do we have questions thank you for great presentation my question about cyber security and privacy of the data learning what's the future of the data learning is it to for example integration with HAI and quantum we do cryptography and something else what else should be done to be more efficient and more private and more secure so yeah thank you for a question that's a very good question I mean the main thing is in terms of like when you ask about privacy we can make it as private as we want the thing with that is what you want to actually achieve is best accuracy or performance privacy tradeoff so that's one of the main challenge because I mean always you just don't need to communicate you don't need to care about any privacy whatsoever I keep my data private never communicate that so that has a perfect privacy but essentially zero utility like the main challenge that people are looking at is right now like how to first of all it actually turns out that it's even in this kind of federated network it's highly non-trivial to even train without any privacy constraint just as I introduced that's still kind of non-trivial task then when you add like formal privacy guarantees on top of that they introduce the extra layer of challenge and there is a lot of work going on on to like how to find the optimal kind of between the utility and privacy so that's kind of the main thing but you can deploy it essentially anywhere where you collect the private data or where you cannot collect private data but where you want to actually train on the private data Hi, thank you again for a great talk I have a question could we expect at some time some I don't know unified framework for doing that like hug and face it will say like combined all this and from the user perspective you just check the model and expect something similar for federated learning because you've mentioned a lot of frameworks and they're kind of independent to each other so what's the reason it's like still under progress Yes, I like pretty much I mean that's the main two frameworks I mean they kind of like where I think I mean they're also both of them like startups and one of the kind of main goal for them is to provide kind of like unified one-stop shop for federated learning that's thank you so much for very interesting talk so my question is about as I understand there are several approaches which helps you to make training process more efficient and you show some graphics on ResNet as far as I remember yes and on ResNet we have approximately no drop in accuracy approximately no drop in score my question is the result of this good score is the result of some of these approaches or you apply one for example drop out or matrix approximation of the layer Yes, so I mean essentially I mean the ones where you do not essentially lose any accuracy I mean what we kind of show here this is the one that you point out to yeah, so essentially what we show here that I mean first of all like if you kind of do things well you can kind of improve tiny a bit yeah, so essentially I mean that's when kind of a regularization kicks in like the standard thing that you would expect to get with the decomposition that you remove a noise and then we show like you can actually like if you really care about are you trying to push the number of parameters smaller and smaller you can still do that by just increasing this like a sparsity penalty Okay so am I understand correctly that this table shows the approach of regularization Yes, so it's I mean regularization plus pruning, right? Dear colleagues I think that we are a bit running out of time because next sessions should already start so I suggest that if you have any more questions to Sam that you just approach him like during coffee break and something like that, so let us thank Sam again. Okay, so thank you very much I going to present a joint work, mostly work has been done by Osge Sevgili so she is a PhD student at University of Hamburg and also this work is with some collaborators also from Hamburg and Institute of Technology so the work is about the task of ultra fine entity typing so what is entity typing and ultra fine entity typing is consider this example you have this Olympic National Park and this is a mention many of you know about entity linking task and this is when you take some mention to knowledge graph knowledge base let's say wikidata or wikipedia so you assume then in wikipedia there is a specific page about Olympic National Park and then it's quite useful to link these because you can harvest, get all the information from wikidata about this national park maybe attributes, mentions, description and so on. Now the reality is even the biggest knowledge graphs due to inherent power law of distribution on scarcity of the data they cannot cover everything so I remember this case then there is this bubble net database like knowledge graph and if you look for word python there are about 50 cents of the word python and there are like two roller coaster parks one in Germany and another in the United States and Florida somewhere and that actually gives you an idea that you even if you no matter how hard you try there will be always this gap there will be always the long tail of some entities which just nobody even either cared to enter in a knowledge graph or actually never managed to insert in there so that's why entity typing come to rescue then you just label with certain hyper name like is a relation and then you can still get a certain idea of what it mentioned is even if it's not inside the knowledge graph now inside this area there are different granularities because you can let's say this is a park or this is a location or different area or maybe you can say this is an Olympic park so there are really also a hard question what kind of granularity you need to take and we speak about a case when you deal with a pretty high granularity so you speak about like tens of thousands of census or labels but then the task is relatively easy when you take let's say 100 or 50 different types then you can collect easily a lot of data but this information might be not so specific not so useful for different applications and the problem with ultra fine typing when you deal with a large vocabulary is then it starts to get again hitting this problem or scarcity of the data so you don't have this amount of data and people try to come up with different ways with this problem and one of the approach maybe not the most successful even but still interesting I hope I will present today unsupervised approach so that we don't use certain manual annotated data rather we use distributional semantics and let's say bottom up approach to do it how people also approach this problem well people try to use the distance supervision for this entity typing what people try to do they try to think hey where can I just get the data for free where did it occur you might get entity linking data sets and for entity link you can look in the knowledge base and see what is the hypen of this entity and then use it maybe generate several hypenings this is a common approach to let's say automatically generate such kind of data sets people also just try to use something like HRS partner so rule-based approach to extract these easy relations from text let's say I say a sentence like such courses Mercedes BMW in Audi they are expensive and luxurious well that would be like a second approach but there are also some approaches which use short unsupervised techniques again to bottleneck what we are trying to do so we are trying to leverage unsupervised induced word senses using this job in text framework which is based on distribution of semantics and it contains not only distribution representation of words but also distribution representation of word senses labels with hypenings and what we actually do in this work we try to see how much useful this hypening labels to the task of entity typing so we try to disambiguate the context with respect to this induced senses so how it actually works so at the core of this approach a repository of this usually induced senses these are coming from this job in text framework and you can look at this paper and this paper just for the background work so this is not what has been proposed in this work but rather like infrastructure which we are using and what it proposes for every word you have let's say in this case is word RANS it might be a city in France so in this case you see this first cluster let's say Lille, Montpellier and other variations of RANS but you also have this football club well basically this football club has a different hypening and here you see these labels for easy labels the common is labeled for one sense and the city is a hypening is a label for these sense and well basically the magic of this is that none of this is done using human labor instead how people actually obtain this first you obtain something like a word to work related words and then you perform clustering and then you group these words obtain these clusters of words but again you don't have hypenings but how hypenings are obtained every word is assigned a list of automatically induced hypenings let's say using these patterns and then these counts correspond to common hypenings for this cluster let's say so of course this term will contain a lot of hypenings but the common hypenings will actually pop up at the top so that's actually the trick how you get relatively clean hypenings here at the top so city or club they will be common hypenings for a lot of these distributional decisions because they are all cities they might be even some noisy words but still but this is what we are kind of leveraging what we are using now how method works well you have this input always you have of course certain mention you need to assign the hypening to in context so you need to disambiguate this and this becomes your sense repository when the rest is very simple actually so you vectorize the context using input representation and you vectorize the mention because it also contains the word rinse and then you vectorize these guys as well using of course the same vectorizer so they can be certain similarity competitions can be done and basically the rest is just picking the most most relevant cluster and picking hypenings from it to label it here of course a few additional steps apply so if you go for real data sets for entity typing you see not just a new word but something like this so mentions might be really long might be really elaborate even if you obtain sense inventory in this bottom up way from the text corpus there might be no sense representations or mentions for this kind of multiple expressions so that's why a lot of additional preprocessing are done so headwords are collected so different keywords from these mentions might be collected and then candidates for senses they are obtained not necessarily from the exact mention of course also certain other steps like singularization of the hypenings and the mentions are done so mention might be done most smoothly but essentially as soon as all these normalizations linguistic normalizations are done well you perform this vectorization of the context and you compare the context vector with candidates prototypes for different senses and you pick the most appropriate hypening label so experiment wise what has been done we look at the setup of choi et al which was this paper and compared to it and to some other baselines the first baseline was just to pick the first cluster for given word or pick a random cluster so this means picking something so this is a very important baseline in every words and disambiguation task it's called most frequent sense baseline why it's important of the distribution again so most words are used in the dominant sense and of course the largest cluster or first cluster in this case will have dominant sense and this is kind of considered always a strong baseline random cluster just pick a random sense and then choi et al well this is approach which rely on encoding with a bad direction last tm cnn and train some multitask objective and also some other approaches based on masculine which model and nli were considered of course there are quite a few moving parts in this method so you can select headwords differently you can singularize these different words mentioned differently here and entity linkin and principle the way actually how do you yield candidates maybe is more important the kind of neural network you use this is my many studies show this because this is a similar study this is not a difference if you do some entity linkin business or this kind of entity typing business the way how do you select mentioned how do you match it it's very important how do you generate candidates okay of course some parameters correspond to job and tax itself for job and tax is a fixed let's say granularity of clustering so clustering can be fine or more coarse grain so you can have for different word let's say for jug word you can have three senses or you can have ten senses and of course they might be also different more noisy less noisy and so on so forth here of course last parameter we will take into account and important one is a number of predictions so you can take only first or you can take more just five or ten here you can just return to this example so first second third labels they seem to be relevant but if you go down the list this is just automatically created list and of course you hit at some point where there will be like a very noisy hyponyms which correspond to some very generic or irrelevant senses all right so here is a table with results so first of all we see that indeed first cluster and random cluster they relatively strong baselines so but the method itself is actually outperforming these baselines so actually it does a certain disambiguation consistently however the results for the literature are pretty strong based on these other approaches so in the end what kind of a contribution of this work managed to do is to in combination with approach of joy the method yields certain improvement so basically itself the methods show that the clusters are pretty noisy so just using the method by itself yield quite noisy results but it yields some additional information and if you combine this with method of joy especially taking only a few predictions so you see here the precision really drops quite significantly if you take the first hyponym or second three, first five, first seven so precision drops but recall of course increase with time so what actually worked much better is also to drop in pronouns what is the reason why if you combine with pronouns if you work with pronouns it doesn't really work well the reason is very simple think of pronouns like it if you have senses even if you have some senses several senses of the word it or she he hyponyms of the senses will be completely meaningless and if you take some mention in context and you have like mentioned it well there is no way to correctly generate candidates for this so that's why if you take setup without pronouns this boost actually for ultra fine setup can be obtained like you see here but that's kind of summarized in this picture if you don't if you consider pronouns the results more or less will stay the same but this additional information coming from distribution labels they're useful in setup without pronouns ok so in this case slight improvement of score improved and of course there are certain errors in the method you can see here this is what was generated and these are kind of two predictions so for instance you see the method generated something like violation difficulty for this mention but the true label was like a crime or something and again you see the task is pretty challenging for real datasets as you have very long mentions and in this case the method seems to generate quite plausible results but again I couldn't sometimes these might be somehow relevant but they're also not mentioned in the gold standard ok so the summary of this talk is like this we explored how information from job in text from unsupervised induced word senses can be used for the task of unsupervised for entity typing it seems that if you don't consider the pronouns results can be improved if combined with method of choice that means word senses contain helpful and complementary information however word senses induced just from text they are pretty noisy and you need to deal with them with extreme care because also you need to be very careful about how you can this generate that's pretty much it in case you have any questions I will be answering and Osge might be also on a zoom link so in case Osge you want to say something yes I am also here thanks for the talk so yeah just one very simple we are done with the English data so I am not familiar with this Job in Text framework but how easy would it be to extend it to other languages and does it exist for other languages yes it actually exists for multiple languages I am sure it has support for German for Russian for Italian not sure how long this list is but yes I think about let's say maybe 5-10 languages are supported and yeah you can look at the website they have a nice web demo you can just enter a word and look up all of these word senses actually it's a snapshot of this demo so you enter certain term select example and language and you will see something like this you see this automatically induced since clusters and these labels thank you very much good morning guys my name is Maria Maslova and today I am going to present a project which title is Rukam comparative argumentative machine for the Russian language and my colleagues Irina Nikishina, Stefano Rebrokov as well as Chris and Sebastian so I am going to speak about quite an important topic the problem of choice I am sure that all of you have faced the necessity to choose between iOS and Android holiday places car models and so on and one more topical issue is the choice between cats and dogs so who is for cats in this auditory raise your hands and for dogs I see so in the end of the presentation we will see how our system responds to this question so it is quite logical to create a sort of system that will help to solve this problem the problem of choice with the support of some reliable arguments however this task is quite complicated as it lies concurrently in the field of question answering and argument mining still one of the most known and prominent research in the field is CAM the system which can answer users comparative input with the support of some arguments extracted from the large text corpus however CAM is English-oriented and there are no analytics for the Russian language so now we present RU-CAM a system aimed at comparing two objects from general domain in Russian with argumentative explanation if compared to its predecessor RU-CAM has following differences it allows to work with comparative questions in natural language it has the component for object and aspect identification from comparative questions and it uses an elastic search index of open stupor-ratch crawl-degraded corpus abbreviated OSCAR we do not only develop a similar system and a pipeline from an engineering perspective we also try to pose and answer following research questions what are the main peculiarities of CAM that need to be taken into account when adapting it to other languages and one more question more specific one what are the main challenges for adopting CAM specifically to the Russian language to start answering these questions let's look at the system design the process can be split into two steps Russian analysis and argument retrieval argument the first step is about the identification of interrogative and comparative nature of a sentence also it includes object and aspect identification process and the second step consists of the search of relevant arguments for the important objects their classification and ranking let's consider each step in detail the processing of a request starts with identifying the question type so whether it is comparative or not it can be done in different ways including a rule-based approach here we stick to the idea of special patterns in comparative questions which include comparative forms explicit mention of comparison, similarity difference etc to implement some machine learning approaches we first compile a data set from the sources shown in the table and here is an example taken from this data set then we use a rotating bird and a fine-tuned bird from another research as you can see in the table the letter shows the best results but comparative questions are quite specific kind of question that can be identified with nice quality even using rule-based methods after identifying the question as comparative we need to extract objects and optionally aspects to further provide them for the argument retrieval stage at this step we also implement several approaches including a rule-based one it is founded on the idea that all requests have certain structure namely they contain two objects and connective of comparative nature between them we consider foreign cases two nouns, two verbs the combination of noun and adjective and the combination of noun and two subordinate adjectives also we expect a connective from the list of conjunctions and synthetic words expressing comparison between these two objects in order to create a data set for the task we take thousand sentences from the previous step that have been labeled as comparative and manually annotate them and the results in computational linguistics were asked to label the first and the second object and optionally aspect and common object common object is a specific structure with noun subordinating two adjectives so for example in a sentence in a phrase green or black t one object so the level of annotation agreement is shown when creating the final data set for models fine tuning we use the notation version supported by the majority of annotation and that's one more example considering our own example we use fine tune transformer in quarters and if you short approach on generative transformers to solve object and aspect extraction task the table presents the results which model we see that generative models perform on power or even slightly better than baselines and significantly worse than transformer encoders still they may perform much better after proper fine tuning as we have shown them only five examples regarding and zero scores for common object and aspect labels we claim that there are two problems the first is the inconsistency of annotation and the second is a complex nature of these labels in semantic sense in order to retrieve documents in favor of one or the other object we use open super large called aggregated corpus oscar we use oscar instead of the common crawl while it is claimed to be its filtered version we store and index this data with elastic search when indexing documents we decide to create two indexes the first one for storing document information like the number of sentences web and so on and the second is for storing sentences themselves two or two sentences we first use we first do the snowballs terming not lemmatization because of the type constraints and then apply wildcars to be able to find all word forms in the boolean gson query and require that the course must appear in matching documents we consider the step to be the most challenging in all camp pipeline as russian language has highly fusional morphology which makes it much more difficult to retrieve sentences than in english because query words may occur in any form and look at the elastic output for our objects just an example after the candidate sentences with possible arguments are found it is necessary to understand whether the sentence argues in favor of the first or the second object again we have a rule based approach that requires a list of keywords with adjectives and adverbs with the meaning of superiority or inferiority of the first object over the second also take into account negation cases when the sentence is reversed we collected data set from 140 ps and annotate them using the yandex taloka system for data crowdsourcing to do this we select same or similar ps from the same domains as in english research like programming languages car manufacturers food drinks and so on and make queries to elastic search to extract all sentences matching the query then we create a system of text butter tag means that the first item wins or the second worse tag means that the first item loses and the tag none means that there is no comparison between the objects we are interested in unfortunately the annotated data set is highly imbalanced 75% of sentences and for example only 9% belong to the worst tag and the study also implement several transformer encoders and few short approaches with generative transformers the results for comparative sentences classification are inconsistent and relatively low for all the models due to the class imbalance problem it is interesting that a real based approach produces a quite decent result on better better better better sentences than it even outperforms both large on worst sentences and generative transformers on non sentences the process of sentence ranking is identical to the one in CAM we score comparative sentences by combining the classifier confidence and the elastic search score when displaying the arguments in the CAM on a certain object we sum up not only better arguments where the current object is the first item but also worse arguments where the object is the second one in the sentence for instance both sentences are better than dogs and dogs are worse than cats are used in favor of cats when comparing them with dogs the main outcome of our research is the final system where we integrate all the parts described above the evaluation of the system is currently working progress we plan to evaluate to come analogously to CAM evaluation pipeline by asking whether users are performing faster searching something of comparative nature if compared if compared the work with CAM with the keyboard search and also we can ask some users just to play with the system to collect the feedback that's it about the pipeline and it's time to answer research questions in general so in general when transferring CAM to other languages you should take the following peculiarities into account the difference in the notion of comparative sentences in different languages difference in the syntax and morphology of languages when implementing rule based approaches and the existence of the relevant data sets and pre-trained large language models for training for different sub tasks as well as large text corpora containing comparative sentences for search in the target language so nevertheless as it has been shown in Russian it might be quite smooth if at least some of the required tools are available what do we have now here we have CAM the first instrument which helps to answer general domain comparative questions in Russian inspired by the CAM system we create a similar pipeline adding new steps for comparative question identification object and aspect identification and sentence classification we also present several new data sets in Russian that might be further used for the fine union of language models for the sub tasks and from the preformed experiments we can see that rule based approaches show decent result on all sub tasks of comparative question answering as well as few short generative transformers and this sub task needs to be further investigated and finally let's look at our comparison of small research according to our system cats or dogs and do that quite confidently so that's interesting and that's an example of some top ranked sentences extracted from the elastic search as future directions we plan to incorporate a summarization system that would be able to produce a coherent answer from two lists of arguments for each object it will allow us to compare the results of various instructions tuned models for Russian and charge gpt with the rootcamp pipeline thank you for your attention any questions thank you for your work can you elaborate a bit more on how you plan to evaluate the system outputs because your experiments were based on different components whether classifier argument identification works well whether sentence classification works well but ultimately a user has certain information need and it's presented with these outputs whether cat is better or not and how would you judge whether the system is satisfying this information need well or not so for users there will be sort of frontend to use and to input and work slower, faster and what else should I say okay yeah that's a human study do you think it's a possibility to do it automatically like a reproducible way so that tomorrow somebody will develop another system and it can be compared as well or it's just not obvious as I understand for now the main way of evaluation is human based but maybe we should think more about some ways of evaluation of automatic evaluation also that just a sort of future work I suppose so okay and maybe okay I will pass thank you for the talk related to what Alexander asked have you tried to ask questions like compare something that is incomparable and how system deals with such thing I mean it's partially due to lack of data partially due to some strange query and what is the default behavior, what is the intended behavior in your system like if I compare cats and Audi or something like this but what yeah of course in this data set if you train it on this data set cats will always better better than humans yes in fact I suppose that will be not enough output from the corpus if we pose such question yeah clearly but what is the default behavior in this case or some like what system should do for now yeah for now it's gonna be done in request so if we identify that the question is comparative then the two objects will go to the elastic search system and will retrieve some sentences in my opinion there will be almost no sentences comparing cats and BMW sometimes we have a cat as a machine and then this is the disambigation problem in this case but mostly it's for future work so there might be some incomparable objects and this is quite limitation for now but in future work normally we are planning to play some taxonomic structures to look for hypernames and looking for how the clothes are related or maybe they're like in the different parts of the graph so they cannot be compared not compare yeah yeah that could be done with some more frontunia something or some kind of unsupervised baselines using taxonomy and wordnet in my opinion thank you for the talk have you considered to do more classes it's like it seems to me that sometimes the answer to the comparable question is things are equally good or equally bad and have you thought of this classes we've never thought about that but I think that as a final result we should receive a sort of process for the first object and the second object and that output requires the classification better of course and none and nothing more okay thank you let's thank the speaker again now it's time for the third talk okay hello everyone I'm Maxim Savkin and I would like to present our paper called tuning free discriminative nearest neighbor few short and end detection their consecutive knowledge transfer so let's start from an introduction to the task intent classification is task of identifying user intent given an utterance naturally it appears in dialogue systems and comes along with the task of out of scope detection here you can see on the slide tell me a joke is an example which doesn't belong to any of the predefined in scope intents so it belongs to the out of scope class I would like to emphasize the importance of out of scope detection as it is crucial for generating an appropriate response we'll solve both of these problems simultaneously so the motivation behind this work is that most of the existing methods for intent classification rely on expensive fine tuning and have high training requirements especially state of the art models and also most of them are focused on in scope classification completely missing out of scope detection our approach on the other hand in our approach we try to create a model which can work as a service so it doesn't require any task specific fine tuning it takes a few short data set and a set of unlabeled utterances as input and produces a set of intent labels we try to inherit the discriminative nearest neighbor architecture it utilizes a standard k nearest neighbors and replaces the distance function with a deep cross encoder repair the model this model takes an input and a training utterance and tries to predict the probability of these utterances belonging to the same intent class so it is some sort of a similarity function but based on a deep cross encoder model so in the original paper strong capabilities of this similarity function were achieved by fine tuning it on pairs of examples from the target data set however we suggest to completely skip the fine tuning step and focus more on creating a strong pre-trained similarity function which can differentiate between unseen and tense so further I'll focus on pre-training this similarity function we consider using several binary classification tasks the first one is natural language inference it is popular for pre-training some strong binary discriminators the second one is paraphrasing it suits a bit better for similarity prediction and the final one is a consecutive pre-training where model firstly trains on a large natural language inference data set so that it can learn some utterance relations and then it tunes on a smaller paraphrasing data set so to better measure the similarity prediction we also for natural language inference merge the last two classes neutral and contradiction into non-indulment so it's a binary classification task one problem with paraphrasing is that it lacks large high quality data sets so to mitigate this issue we've tried using a small high quality data set and augment some non-paraphrases for generating non-paraphrases we use some sort of clustering you can notice that paraphrasing is an equivalence relation so paraphrasing data set can be divided into equivalence classes and utterances from the same class have the same meaning and can be considered paraphrases and utterances from different classes can be considered non-paraphrases they possibly have different meaning so all those missing connections you can see on the slide will become our newly generated non-paraphrases which we will use to increase the results for paraphrasing data as for the metrics for the final model for classification we are using in-scope accuracy out-of-scope precision and recall which are defined as standard recall precision but positive class is an out-of-scope and negative class is a combination of all in-scope classes we randomly sample 10 few short data sets from the original data sets and report the average and standard deviation for all the metrics and we will be using a large clink 150 data sets which contains 10 domains and a wide variety of intents and a banking data set which contains only one domain but 77 fine-grained intents so let's move on to the results of pretraining as you can see on the left plot a natural language inference task despite being so popular achieved the worst results due to its directional nature and the best results we were able to obtain so far are with consecutive pretraining where model firstly trains on a large natural language inference data set and then tunes on paraphrasing data set with newly augmented non-paraphrases this is the first third column I would also like to notice that augmentation really helped to increase both in-scope accuracy and auto-scope recall by introducing new non-paraphrases so let's compare our model against other tuning-free methods the first one is TFIDF KNM classification and the second one is embedding which is actually just a KNM based on being coder Roberto model which was pretrained on a natural language inference task only pretrained, no fun tuning at all so here you can see that on the clean data set and banking subset of clean we are achieving the best results so far even getting a much and our approach is much more stable to threshold selection our approaches as it has a much larger area under the curves of in-scope accuracy and auto-scope recall we also thought that it would be important to compare our model with some fine-tuned methods DNC is the state-of-the-art model for auto-scope detection and here you can see that here you can see that as expected fine-tuned methods are better in initial in-scope accuracy however for standard Roberto model you can see a huge drop in accuracy which means that it has a lot of low confidence predictions and our model has a larger area under the curves of in-scope accuracy so it is more stable to selection of threshold these results are not included in our paper but we decided that it also would be important to see how our models tax against chat GPT so the whole clean dataset didn't fit in the prompt so we've used only a banking subset of clean dataset and with standard auto-scope examples you can see that chat GPT with zero short prompts achieves nearly ideal results and we suppose that it's all due to the fact that it memorized this dataset quite well to change all text labels with indexes and we also replaced standard out-of-domain examples out-of-domain out-of-scope examples with harder in-domain out-of-scope examples and as you can see our model still attains relatively high recall and accuracy while chat GPT really struggles with out-of-scope examples so only a usual chat GPT is able to produce relatively good results yeah, so summarizing our paper I would like to say that we've developed a model that doesn't require any task-specific fine-tuning so it can be applied to any dataset for intent classification it supports an out-of-scope detection has the best performance on clean dataset and it is robust to threshold selection thank you thank you can you elaborate on this difference between setups in-domain and out-of-domain so you mentioned just next this one just the one before conclusion so you mentioned that you removed some identifiers and converted them to numerical what does it actually mean I mean that chat GPT probably has memorized the dataset quite well so we decided to reduce the effects of this memorization and replaced all text labels with just indexes so it couldn't memorize just by prompting so well would it be fair to chat GPT in the sense that maybe your system memorized this indices no, it doesn't take labels just at all so it just takes input utterances and compares them between themselves only and doesn't take a label as input so it's totally fair I think do we have more questions in the audience maybe we have some time for one zoom question if we have one no, we don't have questions in the zoom okay now it's time for the fourth talk hello my name is Vasily and we are going to talk about whether it is possible or not to find the number of topics in a natural language processing dataset so a couple of words as an introduction what is topic modeling about a topic model receives as input a huge text collection and as output it produces topics topics as probabilities over words so we can see what text is about and we can locate places and text where each topic is covered so the problem of topic modeling can be viewed as a matrix decomposition problem topic model receives as input a matrix of word in document frequencies and it decomposes this matrix into matrix multiplication of two matrices first matrix of word word probabilities in topics matrix phi and second matrix of probabilities of topics in documents well, but it is not clear how one should select this hyperparameter is it crucial or not how to find it so it seems that in some text collections at least the number of topics can be well defined beforehand for example if we take some articles from Wikipedia they are labeled split intersections so if we take some articles from let's say art section some from biology section some from history we get a data set and we can say obviously that there are three topics in it so but in real life it may appear more complex because these topics labeled by humans this is something that just helps to simplify the categorization process so in real life it may be well much more topics in this text collection these bigger topics may be split into smaller ones and furthermore topics may combine and produce some new topics so is there a number of topics in text collection or not can we find it or not we are trying to find an answer to this question and what are we going to do we are going to train a topic model for a text collection with varying number of topics from low to high and we are going to track some topic model quality measures and look at the plots, at the dependence of quality measure against number of topics and if we well let's say see some local minimum or maximum or plateau it may be a sign that the number of topics corresponding to this interesting point is an optimal one so this is the picture which we are well expect to see after going further I think it is important to say a couple of words about similar projects we are not the first to try to track a lot of quality measures while training the topic model we are even not the first to try to find a number of topics using these quality measures but we believe that our research is one of the most extensive one so what quality measures are we going to use first perplexity one maybe the most common measure while training topic models the lower the better second block measures diversity measures they compare topics with each other computing distances between pairs of topics because if a topic model produces topics which are all similar it is bad the second block is clustering measures because topic modeling can be viewed as a soft classification problem words are split into topics which are soft clusters so we can adopt several measures from clustering analysis to use it as a measure of topic model quality and the last on this slide is a block of stability measures well topic models are unstable it means that if we train a topic model on the same text collection with different random initializations we can get different results so we compare different topic models obtained with different random initializations with each other compare the topics well this is not all measures which we use the next block are information theoretic measures we use several of them but the idea is the same they compute roughly the difference between model complexity and model likelihood the bigger the likelihood the better but the bigger complexity the worse so these metrics are trying to find the balance between the two the two model characteristics the next one is entropy there are works where the authors propose to use this metric because another analogy between topic modeling and a complex system where we have several possible states topics and particles that is words can occupy several of them so the optimal number of topics using this measure is the number of topics which gives an equilibrium state to this system and the last block which we use the last block of measures are top tokens measures we compute coherence and lift scores that's it well the methodology of our experiment is roughly described here for each data set we train a topic model with varying number of topics from minimum to maximum compute topic model quality measures and look at the plots minimum and maximum topic model in maximum number of topics which we vary depends on the data set the models which we use are the following PLSA the simplest model which have only one hyper parameter the number of topics LDA the most known topic model and a couple more the related topic model which is trained in order to produce topics which are distinct sparse topic model which distinguishes its topic its topics it splits the topics into two groups background topics which are smooth and well about nothing and specific topics which are sparse and exact sparse means that the largest probability mass is spread only on a small group of topic words well this is models and this is data sets we use several data sets in English several in Russian language for each data set we know at least approximately the number of topics in this data set they expected this is our ground truth the problem I should say a couple of words about the last data set this is our data set composed by our research group it consists of good articles from Russian Wikipedia this is the result table with three numeric columns each column is a score which we assign to each quality measure so what do they mean the first column is jacar metric it tells how the predictions of the same topic model but trained with different random seeds are consistent with each other the lower the better in this table it is called jacar because it is computed the following way we take the predictions of topic models with different random seeds and we make these two sets the first set is union of predictions the next set is intersection of prediction and we compute the jacar distance between these two sets the next column is informativity it tells how the plot of dependence of quality measure against number of topics is readable by a human local minimum maximum or plateau or it just random up and down and without any possibility to make a prediction out of it the last column is called expected well it tells whether the quality measure succeeded in finding the exact number of topics whether it corresponds to the ground truth from the table with data seeds well what can we see from here the best values in table are colored with blue but these best values are obviously far from good so it may be an indicator that there is no such notion as natural number of topics in the data set and there are several illustrations to explore some other conclusions first we found that optimal number of topics depends on the topic model used for example on this plot oops on this plot we can see three curves each curve correspond to one each curve correspond to a sparse model with different sparse hyper parameter so one is more sparse and as we can see these models this is the same sparse model with different parameters they give different results as optimal number of topics as local minimum what is more we can see that this well broad lines have reached over random seeds also each random seed produces a bit different number of topics so it is also what is the result and on this plot we can see well not about this plot other finding is the following different quality measures produces different results as number of topics this is general case which was started however sometimes on this plot exactly we could see that different quality measures give the same result for example here one topic model and several quality measures and they point roughly at the same number of topics seven but this is not a rule this is an exception and that is probably it so conclusion we found out that number of topics is probably not a natural characteristic of a data set it is just another hyper parameter of a model and it is also dependent on the quality to find the number of topics perplexity and coherence maybe bit surprisingly they failed to give any decent results however information theoretic criterion and ring entropy achieved best results well as a remark as a final remark we want to say that probably this is not such an important task to find the optimal number of topics finding a way of training a topic model which has all topics interpretably good whatever topic number you assigned at the beginning well that's it thank you for your attention I assume it's fine listening me from the mic here so I have one small question so could you please summarize what we should do if we need to select the optimal number of topics so basically run as many models as possible and then to select as I understood correctly but the best way we think that if you should know beforehand at least roughly what number of topics you have in your collection and it is the best way and the second if you don't know how many topics you have or 200 then this is best to just probably train experiments and in order to find a topic model which best describes your collection another way to train many topic models and collect good interpretable topics put them aside and by looking at how many topics you collected you can see how many topics you have so something like this you start with some number of topics and start experimenting making your topic model better or collecting topics do you think you have more questions yeah thank you did you consider some classification based experiments where you apply obtained representations to measure quality of this no we will concentrate on just intrinsic quality measures without trying to assess topic models by experts or secondary tasks because well if we try to select the number of topics based on secondary task it obviously would produce bad results because we just find the number of topics which give us best results we wanted to find out whether there is some natural number of topics which could be found by intrinsic quality measures yeah I think there might be so different for different classification tasks or for different applications different granularities might be needed so that might be at the heart of this issue right so you have different hierarchy of granularities maybe for retrieval you need coarser, granularities for I don't know authorship or identification or something you need very specific things so there might be a general idea that everything in computer science depend on application and there is just no universal representation I mean at least in terms of topics which works always well like in clustering you might have different views on this data yes yes if you are trying to solve some secondary task it is best to search the number of topics based on this task yeah we have more questions yes we do no no Andrei we have one question from the audience and then yours thank you for the insightful talk I have a couple of small technical questions that data set were weekly good and as I believe in all those data sets you have the true number of classes the topics right so how did you obtain the topics from the rule right so this category is probably but what was the label set well yes it was great from Wikipedia we knew the categories of which article so that's how we found the number of topics I mean what was the topics like some categories from the Wikipedia some large categories yes which are good topics I divided into there are articles which called good which are checked and well big and thorough and there are several categories which these good articles belong to so there was some manual post processing for the label set right yes yes great and second question also the technical what motivated the choice of the models in concerned right PLS ALDE and different flavors of ARTM so I mean the kingdom of topic models is rather large so why these well our main idea was to take just several topic models in order to exclude some well biases towards topic models we just wanted to take more than one topic models so we took PLS ALDE as best as well known as simplest approaches and several other solutions just to to make more I mean wouldn't like no topic models contribute some something else or it was not really that important for the study no we didn't consider neural topic modeling because as well as I understood as well as I understand there is also a hyperparameter tier for these models well we just excluded what we excluded the models which also tried to find this number of topics as a result for example hierarchical direct lead process we excluded these models because they introduce additional hyperparameters which needs to be optimized and they are also known universal because in terms that they are assessed differently than the majority of topic models which have adjust to matrices f and theta yeah of course but some topic models have t as a parameter like ETM for example but anyways thanks for the answer thanks for question let's thank the speaker again and now we have the last talk of the session hello everybody welcome to this talk this work is done in collaboration with Kazan Federal University so this is a part of the project of studying text complexity on different levels so this part is related to text complexity at the sentence level and the other parts of the work of this big project done in KFU is related to academic complexity academic text complexity and also lexical complexity so the context of this project is that we want to do the prediction at the sentence level and at different levels as I already discussed and sentence complexity is one of these well-studied tasks already but in Russian it's not that well-studied and there are limitations for classical measures so we tried not only the classical approach with the features but also the deep neural networks to train and measure the performance on this task one thing that is there are other languages like Italian, English and others that already get data set for sentence complexity in Russian there was no data set and brief discussion of the work people tried to do before us and they collected data in different languages that trained some classical approaches like on lexical features and syntactical features as well and deep learning methods like pre-trained language models and collected a lot of different data sets in this domain including English and Italian and our first part of the project was collecting data in order to run the evaluation so the data set we resembled the methodology for English and Italian data set collection so we used the TOLOCA crowd sourcing platform and asked TOLOCA to annotate sentences according to seven levels of complexity but we also wanted to experiment with different features so we sampled the data set from the syntagruse corpus there is syntactic annotations and other interesting things that we can make use of later yeah, that was a sample of the sentences was related to the frequency of the lexical sounds in the sentence so we tried to sample the sentences of average frequency not to sample to complex to rare words and so on as our colleagues before did and it each bin contained 200 sentences so overall it's like 1,000 100 sentences so it's just a sample interface of the UI people just were asked to pick one of these scores there was a 10 assessments per sentence and just an example of a sentence we collected data from people who are native speakers there are no other restrictions or something like this we collected the data there is a slight imbalance of the data here you can see the distribution of these scores in the data set and yeah it's biased towards complexity more complex sentences which is a kind of contradictory to what we have in two other data sets English and Italian but it's still interesting observation regarding the assessment of agreement so we have this distribution of distribution of complexities corresponding to the so the x-axis here is the sentence length in tokens and y-axis is the average score distribution of average score and yeah of course there is a clearly the correlation between two parameters but also we measured the average number of people who agreed about the score on the sentence so out of ten people assessors 4.3 on average agreed about the score the complexity score of course the sentence length is a crucial parameter it's important to assess the complexity or the readability of the sentence but it's not the only one and previously we analyzed the previous work we analyzed what features can be important based on the features that are more important more related to the target we build a simple linear regression and then in this work it's kind of a step further we try to push the performance more to get more quality of this data and then in the second part of the talk there will be some modeling like classical and deep learning approaches here you can see the difference between discrepancy between Italian and English and Russian data and you can see that in Italian and English you have a relatively small number of complex sentences not complex here but complexity on the y-score but on the x-score again just length of the sentence two data sets there are relatively less number of sentences with a long relatively less number of instances with long sentences the same you can see here is a distribution of the complexity score so pink is Russian and this one is English and this one is Italian so the average complexity also different on different languages so it can be due to sampling of course but maybe it's just also related to some other like linguistical properties such as average length of word average length of sentence and maybe the also annotator here you can see the distribution of these properties like this Russian English and Italian data set and here the frequency of lemmas sentence length so it's more or less similar so here the discrepancy is appearing but because we got the logarithmic frequency so after the normalization they look the same so these three data sets are comparable and you can run the experiments several models on this data simple approaches like based on linear regression and decision trees in the CBMs based on classical like TFID of matrix features and then we try to also like modern approach based on BERT model and fine-tuning of BERT and also graph-based neural network so that was the initial idea to select features three features that we can select from the feature set of the and then build the linear model with the three parameters gives quite not very nice results and for this for this quality like RTL score is quite low and MSC and MIE quite low so the next idea about BERT is quite obvious I don't want to stop more on this just fine-tuning pre-trained RU-BERT Italian BERT and English base model regarding the GNN we make use of this model that gets the syntactical tree dependency tree augmented with additional edges and also we we use the features of this nodes in each syntactical tree in dependency tree the features provided by fasttext so it's the convolutional model actually gets the features and applies the graph-based convolution in order to find the representation of the whole of each node and then we do the pooling over the all these tree and the final linear layer decides which what is the complexity of the sentence results of fine-tuned BERTs of course it's as it was expected they were quite decent and this is just number usually low numbers good numbers because of large number of epochs when we compare it to GNN SVM and other models we can see that BERT-based models for all languages are much better and linear regression sometimes just doesn't provide any reasonable result because of maybe a number of features who knows and this GNN model actually is not that bad compared to some other languages maybe that's dataset is available and we are going to continue these studies in both cross-lingual, multi-lingual setting maybe building a model that can be applied to several languages and there is an interesting direction of research when the other group working on lexical complexity so they want to measure the complexity of the word in context and then using this you can measure the complexity of the context itself not just simple some sentences with words but simple sentences with the same word and analyze the complexity in context that's probably it if I have time I think I have some time for questions please go ahead I have one small question why English and Italian dataset use seven scale why do they use seven scale? instead of five for example for me five was the simplest way to think if you are not satisfied with binary classification because it's not the type of binary classification task why not five? why did you decide seven? that's a good question actually I don't remember they explained it well in the paper somehow it's called a Likert scale and that's usually from one to five and in this case it was like seven for some maybe more fine grained is better I don't know really thank you for the great talk and for the dataset finally and the question that I wanted to ask is when I put myself in the shoes of the annotator yet again about the scale well it's sort of hard to choose I believe however you explain each button so I wonder if there are any gamified things or some procedures where people annotate the complexity of the text implicitly so that the dataset can be derived from the procedure is there anything of this sort? something that comes to my mind is this tracking thing that people like measure something like indirectly like how long time how many times you have this circuits or something like this and when you read the text and there are measures and also I think in KFU there is an investigation of this part how students in school read the text and how they perceive it using the eye trackers devices but for this crowd sourcing it's I think the only thing to do is to increase number of annotators and somehow control the output quality like measure the time how much time people spend on each sentence for instance to annotate in each sentence it should be not less than like several seconds maybe so that's kind of general approach thanks okay we have one more questions in the zoom yes please yes thanks for the talk so my question is also about the data set and also about putting myself on the shoes of the annotators so the obvious strategy for an annotator is of course to just label long sentences difficult and short sentences not difficult so the guidelines for the annotators did they include any specific instructions about taking into account sentence length or not taking into account sentence length yes good question yeah there was a short guideline actually we provided examples of sentences hard and not hard and we completely rely on the intuition of annotator in this case I think any guideline in this in this part when you try to gather an assessment of the complexity that is or difficulty that is purely like perceptional thing it's not kind of cannot be formulated or defined objectively in my opinion so we rely on the intuition linguistic intuition of assessors and we minimize these guidelines as we can and then again just if you have enough number of assessors per sentence on average you will get a good assessment good score for the sentence the guideline was very short and the guidelines didn't mention the sentence length in any way no no no it was not related to any bound to any parameters or whatever so we just didn't want to push assessors or bias them towards some attribute or whatever ok thanks ok I can't oh two questions ok yeah thank you for the talk I have one small question have you considered any relation of text complexity with linguistic acceptability if there is some connection between these two notions maybe the text complexity could be assessed with the perplexity of the models and other methods used for acceptability no we didn't measure this exact relationship between these two things what I was thinking about the measure connection between the grammaticality or the some syntactical features that's why we're looking into syntax rules you can compute some syntactic based parameters from the sentence but like grammaticality or some you know quality of sentence can be related to complexity of course but acceptance in this case well I don't think we have a lot of data in russian maybe I'm just I just don't know about the there is the Rukola dataset so it's maybe possible to do this it's a good idea it's just a very short comment related to Andrei's question do you think it makes sense to when annotators are given different sentences to provide the same length of the sentences so that they they never biased indeed by this very strong maybe a modeler can learn better if you provide all that yes that's good point actually it needs different scheme of annotation you give two sentences with the same length and decide ask annotator to measure to answer which one is more complex maybe they pay attention then to more non-trivial features similar thing we tried in previous work that was a classifier that was trained on the pairs of sentences of the same length but for annotators here we collected only the scores per sentence that could be biased towards the length of the sentence to avoid this yes we didn't do this but the idea is quite obvious and maybe we will try to do it right now the goal of the project is to develop a model that will just sample complex sentences or simple sentences thank you thank you hello everyone my name is Alexey Andutov I'm from Manosov Moscow State University I am my colleague Natalia Ukashevich I'm an appropriate researcher with the topic document level relation extraction in Russian what are we going to talk about this today first of all why is the task of extracting information and sub-task relevant we will short-introduce what is the relation extraction task and what is the difference between sentence level and document level relation extraction and I will show the problem of nested entities in this task and what models for document relationship extraction are used traditionally one of the most important task in the NOP is information extraction and one of these important sub-task and least export is relation extraction this task has a broad range of applications starting creating and updating knowledge bases like Wikipedia and WorkNet to structuring documents like why this work is important first we address extraction with nested entities and ignoring these aspects without information loss and next we will focus on the document level relation extraction it's a complex problem and it's crucial for understanding entire document not to be no studies focusing on Russian language have been published yet what is the task of relation extraction look at this example given a sentence the previous cance film festival was held from May 19 to 26 given as entities festival and date we should predict the type of relation point in time in this case and also relation can be formulate as triplet subject object and relation type but in this work we can see the document relation extraction document level and let's look an example about Konstantin Rubinov we can see the two entities music band Verzdonsk and Baronova and Konstantin Rubinov at the same time we see that Konstantin Rubinov same have some mentions like Kuzilio that is nickname of Rubinov and we should recognize relation type founded by and recognize evidence supporting sentences that can help you to understand the relation type in this case and what is the difference between sentence level and document level first of all in first case you have single sentence and usually only two entities but in document level relation extraction you have one entire document entities can be mentioned in various forms and you should also predict the evidence we use narrow data set each contains annotations of entities and relations between some words there are about 30 different types of entities like person or place and about 50 different types of relations like workplace alternative name or work as and importantly that some of these relations occur between sentences in the text ok how this task of relation extraction can be solved the baseline approach to its classification task for a pair of entities in text for example giving a sentence you should mark two entities and apply some of crucifier but we have some problems with more advanced approach for example when we mask our entities we have problem with other weapon entities like Moscow State University and it can it can be used in some models like Spunbird ok and one of these approach is this problem and it's about tagging in my previous work I presented that this problem can be solved by several approaches first of all decomposition into sub tasks and each solved by a separate model and the main approach is joint extraction where you use a single model to extract in your text and in short that our results that using a single model provides better quality this allows for incorporating all knowledges and into one model ok we took this into account and consider it some approach that addresses relation extraction in a general sense and what approach did we use to solve the task of at the document level we started from baseline BLSTM that was mentioned in doc red paper text with features like Gov and entity type vectors and coordinates of entities in text applied by BLSTM and we aggregate some of entity mentions and classify by multi class classification layer but in this task it's important to consider all of entities during prediction and it seems a good idea and in approach of document after encoder our text we build a relation matrix and use uned architecture from computer vision approaches and finally we can pass this matrix and classify the output in addition we previously mentioned that we previously mentioned the concept of evidence or of supporting sentences that can help to recognize the target's relation in our task and in the extended narrow dataset such weblin is also provided the dream approach allows us to the additional markup beyond predicting the relationship let's also model the importance of the sentence in this text and finally we can keep only the most important sentences and make predictions based on this document and this approach is called fusion approach so we see that best result was from dream and fusion approaches using Berta large encoder and also we look at some main problems we encountered when working with a model on the Russian dataset in the rail first of all in this task you shoot process entire document and we use window based text processing with open windows okay next the second problem is that the quadratic complexity based on the number of entities and this is because the number of relationship in the document is the square of the number of entities and a lot of relations in your text is label like no relation and it's important and in cases when are using some large models it can be a help that remove random negative relationships up to set ratio within the document okay finally we conduct first metrics on document value relation extraction in Russia and specifically in the narrow dataset also metrics achieved the state of the art models for English I mean benchmark do credit we explore the issue of nested entities and implemented some enhancements to process longer text and also we see that balancing negative relations helps optimize the training process in the task of document relation extraction okay that's all any questions your experiments were done on only narrow dataset how you would estimate how generic this model is if you just go and start applied to new text let's say from newswire or internet websites just for harvesting very large database of relations about persons would it be accurate or the main shift would be very severe mm-hmm yeah I'm sure that it's a general approach for relation extraction while we use these models first of all we checked all the models that have the best results of the English benchmarks like do credit we used approach that can be used in another dataset like for another language like Russian so I think that it can be used for any domain or any area but in some of cases you can see some problems like problems with I mean that you should optimize your model to a process for example in real time and so on okay thank you very much second question okay let's thank the speaker again and now it's time for the second talk of the session hello everyone can you hear me so hello once again my name is Maria and today my colleague Lona who is here via zoom online and I we are going to present our presentation on the way to control but text summarization however today I'm more like a presenter because the main research was performed by Alona and she's the main contributor so all credentials to her and I was more like a tutor and advisor unfortunately couldn't come today but she's here with us and after the talk she will be ready to answer your questions with me so the application of the new promising approach namely the hydro some approach to controllable text summarization in Russia and the structure of the I'm sorry okay so today first we will talk about the motivation then briefly discuss all automatic text summarization approach and then I'll give the brief overview of the original method and then we will switch to our research talk about the data the experiments and of course we will discuss the result so basically the concept of controllable text generation brings an additional layer of flexibility and customization to the summarization process and these controllability allows users to specify specific attributes of the desired text such as length or style for example this customization enhances the user experience by providing summaries that align more closely with the information and individual preferences moreover it allows users to specify the level of compression to make it easier to summarize the desired text as for the objectives of this study the main one was to investigate whether the multi encoder architecture utilizing transformer based model called the HydraSum is applicable to the Russian language because it has shown great results for English and whether HydraSum could produce more stylistically diverse or higher quality summaries than the classic approach of fine tuning language model this research lies in the field of natural language processing with the focus on automatic text summarization as traditional methods of text summarization are mainly divided into the two big groups namely the extractive and the abstractive ones they provide no controllabilities producing texts which are not stylistically diverse the HydraSum method on the contrary introduces some control and briefly speaking HydraSum is a mixture of expert architecture with multiple decoders which is based on the pre-trained language models as a base model the authors of the original choose facebook's bar large but they claim that it can be applied to any transformer model in HydraSum architecture the base model is extended to consist of multiple decoders namely the authors experiment with two and three decoder architecture where each decoder captures different stylistic features of the input text and each decoder has a total number of decoder block where the parameter of the bottom layer are shared among the coders this is done to minimize the number of additional parameters introduced in the model architecture the top layers of different decoders are pre-trained independently thus each decoder can specialize and learn distant representations to suit its specific task and an important thing is the gating mechanism which is an important part of HydraSum basically speaking this gating mechanism is a weighted sum of the k decoders output and it dynamically determines how much each encoder's output contributes to the overall result enabling flexibility in decision making based on the weighted contributions after utilizing the gating mechanism the outputs from the shared layers are fed into a feed forward by a softmax activation function which outputs the overall mixed token probability thus these process assigns ways to the outputs determining their relative importance the author provided three inference strategies namely sampling from individual decoders where one decoder is more extractive and the second one is more abstractive and mixture of decoders using the gating mechanism mixture of using manually specified gating mechanism basically speaking to adapt HydraSum architecture to the Russian language we've chosen the classic summarization dataset known as Gazeta dataset and recently it has been one of the most popular dataset for summarization tasks in Russia the dataset sees soft news articles and there are summaries from Gazeta news website titles of the articles is date, URLs and additional information we also introduced two additional binary column namely the gate column and the sand column as a base model we took M-BART model which is a multilingual language model pre-trained on the massive corpora and besides M-BART we also trained three baselines namely the standard fine-tuning of the transformer model like root C5 base, root GPT-3 small and fine-tuning of M-BART itself because of course we wanted to compare just the performance of the fine-tuned M-BART with the M-BART incorporated in the HydraSum architecture to compare the performance we evaluated it using the classical metric namely we used root scores measuring the quality of generated text with respect to reference summaries apart from root scores we also measured generated summary relevant metrics such as obstractiveness specificity, length and readability just in two words obstractiveness is measured with the help of two additional tools namely coverage which counts the proportion of words presented in both input text and in the summary. Density which counts the average longest continuous extra copied from the input text and this metric for evaluating generated summaries was suggested in the paper newsroom a dataset of one million of 1.3 million summaries with diverse extractive strategy to measure specificity of summaries specitailer 2 was used and the length of summaries is measured by two additional metric its absolute length and the compression rate. Now on to the results we see some significant difference between the performance of two individual decoders the first decoder which is called decoder 0 here provides longer summaries and has lower coverage and density then the second decoder it is called decoder 1 here and decoder 1 is more extractive than decoder 0 and shows bigger coverage the most extractive summaries were produced by root 5 and embark and most abstractive appear to be the reference one showing the low coverage density which is predictable because this is the true answer so mixed of decoders produced more abstractive summaries than each decoder individually and in terms of specificity all generated summaries have results which are quite close to each other it can be explained by the fact that all models will fine tuned on the same data set and therefore they share the same vocabulary however individual decoders of hydro sum architecture have also shown different results on specificity metric decoder 0 has generated a summary with the lowest specificity score among all summaries whereas the highest result on this metric were shown by mixture of decoder and root 5 based model to sum up in this work we studied the application of the hydro sum method to the Russian language we found that the first decoder in it is more abstractive the most extractive summary were provided by root 5 and mixture of decoder provided more extractive summaries than other models and all models showed close results on specificity metric the time is up and practically finished during our experiments the hydro sum approach proved to be quite promising for the Russian language and as a part of the future research we plan to train this model with decoders and on a big and more diverse dataset to try capturing more stylistic features moreover it is important to try manually specify the gating mechanism during the inference stage so now my colleague Alona and I we are ready to answer your questions Alona can you hear us are you here this is Alona hello we have questions thank you for the talk my question about these metrics that you use for evaluation of decoders behavior like naive question is it possible to somehow adjust the behavior of the model or fine tune the model towards more abstractiveness output more specific or something like this as I understand from the last table you just measured this metrics but is it possible to change the metrics the output corresponding to the required criteria is it clear Alona can you answer yeah in fact yes it's like the future work which we are going to do when you are manually specifying the gating mechanism you can do it yourself so you can assign weights and thus you can kind of change the output to be more abstractive or more specific one yeah by adjusting the weights by assigning the weights yourself and yeah so it's manual you can do it manually okay thank you my question is about the mixture of two decoders as I have seen from the results table the results are lower so can you please elaborate how this mixture of decoders was done maybe I missed it and why do you think the results are lower thanks just a second yeah so the mixture of decoders it's when the modal kind of it's when the modal decides which output to generate so it kind of samples from both decoders simultaneously so the result was lower well I think I may add in the mixture of decoders we basically assign the mixture of probabilities from the two decoders thus we basically obtain the weighted probability from the two decoder and here as I remember we used just the averaging from the two of the probabilities from the two decoders and thus maybe we should have carried out and this is our future plan to experiment with different with different decoder weights so that for example the first decoder or the second decoder is more important in the result yeah thanks for the talk just a quick question about this mixture of decoders what about it's computational requirements I guess it's more compute intensive than just using one decoder so how much of a problem it is yeah so I guess when you are using a mixture of decoders when you sample from two decoders instead of one right it means that yeah I guess it's more compute intensive and so so what's like is it like twice as expensive or the dependency is not linear so just how much of a problem it is will it be like twice more expensive well it is expensive it it took a long time much longer than training with one or yeah with one decoder and but it's not really a problem because I did it in my co-op notebooks so yeah I kind of spent some resources on it but it was okay to do that well I guess it depends on the size of the training data or the inference data but just my question is essentially is the dependency linear so is it just the case that when you use two decoders when you sample from two decoders it takes twice the amount of time or compute resources as when you sample from one decoder or is it more complicated because as we mentioned in the beginning they have the shared number of layers the bottom layers they share the bottom layers and that's why of course the inference from the two decoders is not twice as expensive as sampling from one decoder but I'm not ready to write the exact dependency it's much more complicated than but it's less than twice because they have some shared parameters just for this purpose to save some computational resources during the fine tuning and the inference stage Hello everyone my name is Anna and my work is dedicated to machine translation for Russian HAKAS language pair my goal is to present the results that we were able to achieve and also to try to explain how we did it so that you can maybe repeat it on another language pair of your choice so the HAKAS language is a language spoken in Russia by about 40,000 people and have a very limited amount of digitized data so it is considered low resource actually there exist 60,000 pairs in Russian and HAKAS and they are from TIL Corpus of Turkic languages you can see the results of training the baseline model exclusively on Russian HAKAS data and as you can see on the picture everything that is less than 10 is hard to get a sense of and everything that's above 50 is considered high quality and good translations so the basic approach to improve the results of the model is transfer learning which means you initialize the weights randomly and then you pre-train the model on resource-rich language pair and then you take these weights and initialize another model with them and fine tune it on a low resource language pair so the biggest question one of the biggest questions was which language to choose for pre-training we wanted it to be Turkic because we thought that maybe it will help the model to train well and we also wanted to be in Cyrillic script because we wanted to use the shared vocabulary between the parent model and the child model so here you can see the languages that meet these requirements and the sizes of the available corpora so as you can see Kazakh language is the largest one but the problem about these languages the data is mainly web scraped and it is not of a very good quality in terms of translation but also it tends to lean to certain domains like news or government documents so we decided that we will stick to Chuvash language mainly because it is manually aligned and checked and actually it is the second in size so for quantitative reasons as well the preprocessing of the data didn't include much for the parent data because it was of good quality and we just did some additional shuffling and the child data from the till corpus it contains some trash symbols random numbers but also the Russian sentences were of good quality but the Khakas sentences for some reason they lacked punctuation at all and also had some mistakes but luckily we had another corpus that consisted of 30,000 pairs and they were of good quality and also they were the same sentences that were in till so we replaced the translations that we were able to get from electronic corpus of the Khakas language and thus we improved the quality of half of the data set the next step was tokenization and this was done by byte pair encoding with dropout for those who are not familiar with the technology I will briefly explain so it goes like you split the sentences by characters and then you set up the amount of merging operations so you first glue together the letters that appear most often together in the text so on the left side you can see the traditional byte pair encoding and the dropout technique actually is the same but you just skip some merging operations for example you can see on a picture B that on the left and in the middle the RE are glued together and on the right one the RE is skipped and it starts with AT so this is to kind of augment the data because when you split when you do the dropout several times on the source data and get different picking nice sentences you can then assign the same translation to all of them and you kind of increase the amount of the data you can train it on so here's the short information about the setting that I used, I used transformer model in its classic way from the original article I optimized the biometric and I used the shared vocabulary between Chuvash and Hakas data the experiments I did was the increasing the share of the Hakas data in the vocabulary from being the same amount as Chuvash to 1.5 times larger another experiment included chuvash in byte parent coding and the third one was adjusting the maximum sequence length parameter because it turned out that 99% of the parent data was about 100 tokens long and the child data 75 tokens long so we played with this parameter a little bit and actually it showed that all of these experiments combined show the best results in metric