Shall we get started? Yep, it's done. Cool. So welcome to the next network service mesh meeting. So before we get started, if you have anything to put onto the agenda, please bring it up or add it on and. And we'll see about about getting to it. Also, please add yourself to the attendees list. So the attendees list is really useful because. When we go and do outreach, it makes it very easy for us to say that we have a community there that people who are involved, that's diverse. So if you're able to please add yourself to the attendees list. The meeting notes have been posted onto the chat. So feel free to open up there. And with that, let's go ahead and get started with my agenda. So we have. Three recurring meetings. We have this meeting every Tuesday at this time, the NSM doc meeting, which occurs every Wednesday at 8am Pacific time and the NSM use case, which occurs every second, fourth and fifth. At 8am Pacific time as well. There will be a use case meeting this coming. Monday. So. Parking calendars for that. We also participate in the CNCF telecom user group, which occurs every first and third Monday at 8am, which rotates with the use case. We have Shanghai. Cube con China coming up in Shanghai. And the client I will be giving an intro maintainer track. So there is a CNCF answer bar that will also be on. I, I suppose somebody will be. Manning it from NSM from 1132 12. That's me. Cool. And we have a. Tug birds of a feather kickoff, which is going to occur on Tuesday from 11 to 1135. Actually that, that time seems short. We should double check that because last one was an hour and a half. And this one's showing only half an hour. Yeah. Yeah. Yeah. It's really short. That's. Yeah. Okay. And 10 minutes after that is our talk. So. So if you're going to be in Shanghai. Come visit us. We also have a dbtk user space in bird of France. I've submitted a call for paper discussing NSM plus the dbtk integration. The notifications for them should come out. Sometime this week, perhaps. So we'll see. These things sometimes run a bit late. We have a new, so we have something called open core summit. Can someone discuss open core summit? I haven't heard of this one. If someone's discussing it, you're on mute. I am not super familiar with it. We got a direct message on Twitter from open core summit saying, Hey, it would be great to see you there. So there's. I guess you're welcome to learn more on Twitter at open core summit. Cool. So. In my scenario, the main problem is it's probably going to bill. It is during the same time as dbtk dbtk user space, but. Yeah, let's take a look at it and see if there's any possibility we can get someone to, to attend. Yeah. You know where it is. So I'm from school. That makes it much easier. That definitely makes it much easier. So. We have. Oh, and that's Europe coming up. And the call for paper for that is closed. And I believe the, the. Submission should either come out. I think the end of this month or early next month. In terms of the entrance of this, this like that schedule. That will be held in Antwerp. We've had some questions, which the call for paper closes on July 1st. And I believe Nikolai is supplying for a talk there. Yes, I will do probably. We'll see what you figure out. That's still sometimes. Yeah. So for all and has actually. Yeah. We submitted. Me and Taylor and then also. as you probably know yeah I mean maybe we want to track this and see yeah so I put in I put in a talk on crafting of CNF yeah I hope that that one will be accepted to be sure in nice to share our experience and probably get some feedback from what other people think on this and how they see this moving cool yeah and not just crafting a CNF but crashing crafting it with best practices so yeah so we have so we have three other talks then the testbed with Taylor the kernel forder and a possible SMI SNSM integration oh so in terms of we have KubeCon call for papers coming up the closes on July 12th so we need to get a shared document and start gathering ideas so the KubeCon ones tend to give us a lot of mileage and so if you could only attend one that would be either KubeCon would be high up on that list if not at the top we have also there'll be co-located events we know that there's a convoy con will be will be there as well I intend to put something forward on envoy in that particular talk or in that particular co-located event and I'm pretty sure there'll be there'll be a range of others that we can participate in December 10th and 11th we have edge computing world coming up in Mountain View the website is still to be to be announced and we also have Edge Congress Prem I think I saw you want can you tell us about Edge Congress yeah I just got this mail from them saying that this most mostly focused on edge so that's where I posted the program also seems to be relevant more from the edge computing perspective they are open for proposals probably we should submit a proposal so with that if we have any any events that have been confirmed please add them to the website with a pull request we had so on to the social media community team so Lucinda you have the floor good morning great we had a nice post from VMware open source yesterday so I linked to that there and in the last week 15 more accounts followed and service mesh we followed 61 more accounts and posted 23 times I posted about the Cisco live event the kubecon china saved the date and some CNCF events as well so we already talked about the open core summit dm there's also a mention question from someone and I believe Frederick may have responded and continued on the conversation about that no so that's there if anyone would like to continue the conversation I haven't seen this one yet oh I haven't either this is cool thank you so much for bringing it to our attention sure you're welcome so that's linked here in the notes and there's I also noticed a call for demos in the kubernetes community meetings those are held every thursday at 10 a.m pacific time I believe they have things on their calendar maybe through June maybe mid july so if there's if you'd like to do a demo there's a link here for the kubernetes community so I was directed here also because I reached out for this what was the name of the event that we wanted to do the webinar yeah so essentially I think it was Chris telling answering me that because we are a sandbox project we are not supported for a webinar as a project we can do it from the company perspective like VMware or Cisco or you know some of the companies that are participating there and there was essentially saying if you want to do something you can I think that that everyone of the maintainers should be there on the thread you can do this community meeting demos or something like this so yeah it's a thing that's that was the point that makes sense so we'll just we'll just have to make our own webinars in the in the meantime great and last week we reviewed the ovs orbit podcast with interest in giving an interview are there any updates on this possible event we have a scheduled interview with Ben interviewing Nikolai and I about network service mesh it's going to be second of july right I believe so but I don't think he'll release it immediately I think he's trying to build up content so he could really yeah yeah yeah yeah okay so this is where the recording should happen fingers crossed and then yeah well that's great congratulations on getting on the schedule thank you so and on the webinar just to share my experience with the service everything went really smooth really quick I mean like in an hour it was clear that yeah I mean there were a couple of people involved sending responses back and forth and in the end it was clear what what we should do so just to share that if we need it it works really well the service desk I mean great is there anything more on that or should we jump straight into the code of conduct I think that's all from my side yeah great so a couple of weeks ago we brought up the code of conduct so have people had a chance to review it or were there any major issues that popped up in people's mind this is I mean I looked at it before the meeting last week and was pretty happy with the cnc of code of conduct but I wanted to make sure that other people had a chance to see if there were any things that they thought we should do differently and we did get did we get a ping back from me she was saying that that you know from letting her know that that we would be another another group of people through and mediating for yeah I I think I think that the initial request she was a little bit confused because she may have thought we were actually asking for services um oh yeah they can't be confusing we suddenly people don't want anything immediately from you exactly especially especially in this area so so we've we've notified Michie that we were uh that we're considering uh putting adding it on and so she wanted to know as well if there's any problems uh with um with the wording or anything like that as well because the the wording can be changed or clarified as well so if there's any if there's any problems with the code of conduct that we flag they are there are something we can bring up with them or bring it with Michie so cool shall we go ahead and do this then um yeah I'm I'm up for I'm up for for adding an end as are there any how do we want to do this so what should we do a vote do we um so I do have one to ask if you before we merge it if we could remove the work in progress from the PR that would be great yeah all right my ask is to to have the formal approvals of uh everyone on the maintainers list okay um hi Bruce yeah I mean you have the like you have to submit to the PR I mean like I see oh hang on that I see your point yes there's actually a button for it yeah on the formal approval now with the button and everything one other thing I will suggest to folks look into with the with the work in progress tag there is actually a new github feature called draft that will let you push a pull release in the draft mode which literally precludes it being merged until you click the publish button and and that I'm finding to be super cool um because I'm working in other places where sometimes people are super happy to merge the thing you pushed and it's like no stop stop not yet okay so we need one more than um um Andre can you um add your approval or rejection sorry uh yeah okay check yeah don't let us bully you into something as well if there's something that you don't like about it you know definitely and I will change the work in progress right now so I'm not able to to vote on my own ploy request so I'm going to add I'm going to add a comment on it showing my approval so while we're waiting for that last approval let's go ahead and jump into the to the next topic so we'll come back to this can you click on the back lock just it Nikolai has the far um okay so uh the the situation here in the on the release hasn't changed that much since last week except for um we now have the two docker hub uh repos where we can push different images and I guess that it's going to be a manual process for now like publishing the docker images for the release um I can really just talk and so I'm in the process right now for the right of that actually we'll actually use the network ci repo because right now oh we do have the repo but it's not being used and I'm chasing down a few bugs on that PR there's actually a link in the meeting agenda pointing to that outstanding action item so once I've got that so it will actually merge then we can merge it and merge it also to the branch and then we'll start publishing to the network service mesh ci repo for the ci artifacts and then from there we can look at um actually handling proper publishing the way we discussed for the main repo and I think the things we discussed for the main repo was we would have things tagged as releases we would have a latest tag that pointed to the latest release artifact we would have a master tag that points to the latest stuff coming off master and then we would have for release throttle branches you know something like you know branch dash v zero dot one that points to the latest from that branch that was my understanding of the last discussion um that's what I'm trying to do in the PR if I got in the wrong just comment and we can get it fixed so can anyone hear me I don't hear anyone yeah we can we can hear you yeah yeah but but but I was muted and talking to to the okay that's good it was like something I lost with Sutton's Nikolayan but yeah so like I said I still need to be done this a little bit we don't necessarily do this live if people don't like the the the tag choices I'm making discussion okay I will defend the death the fact that it was a seeing choice I will not defend that it is the right choice um so yeah yeah it's a good it's a good start in any case I mean we need to to to move this forward we actually should have moved it much further by now but okay and this probably can also go into the release branch and should go there before we okay yeah better better to get it slow and right rather than rushed and wrong oh yeah that's trade-offs indeed um so the thing here was okay and the backlog so on on on the backlog things things are mostly frozen so probably we need to why is this one fixed that's wrong uh that's probably that's probably uh auto tagging from yeah yeah so I I was looking at some other projects I don't remember which one but people were having tags like near release next release like more generic tags that actually can somehow and this is a slide how to say in tradition to the next topic something that like you can reflect your roadmap in your issues like if you see what I mean like that that's actually interesting um it has the benefit that when that the stuff that you didn't actually get to um that was scheduled for next release if it doesn't get done next release it's still next release yeah exactly I kind of like that it's it's it's very low low effort in terms of going through issues um so I do like that approach yeah um I mean uh like we have a couple of things which are still pending from 0.1 which are obviously not getting there because the branch is there and we are not doing active development and we need to move them to the next release if we are planning for it or we need to kind of put them in some categories saying okay this is near future this is distant future and blah blah something like that although from what I can tell most of the things that we have on our list I mean not not not only here but in general you know the issues should be more or less uh near future um there are very few of the specs where we are kind of discussing conceptual things which might be implemented somewhere down the road but uh yeah as we as we were kind of discussing this uh where was the the link here for the roadmap I was expecting to be able there's a there's an attempt to get a symptom technology tree oh yeah yeah yeah and then there's the spec board because I don't think I want to be super clear the technology tree is probably not capturing everything yet yeah yeah and and so if you're sitting there going where's my spec the answer is um please let's get it worked in there right because I know we've got lots of people doing lots of interesting things in the in the spec space about what we could go forward and not everything has gotten as much love as I would like um because the world is full of shiny objects yeah one thing that actually I would I would love to wrap my head around a little more and it sounds like from the talk proposal Donna is looking at this is I I've glanced at SMI and I like a lot of what they're doing I'm not sure how that relates to network service mesh but I agree with the intuition that maybe somehow and so you know as if on it gets you get closer to sort of figuring out something that makes sense there I would love to see a spec for that um just so we can get a sense of how we might you know double tail end up bigger communities yeah I completely agree with that like I can I can see areas where they can potentially integrate with each other uh vertically but uh you may you may also have insights uh that because we're thinking more about it than than we have at this time so definitely definitely bring you up to us uh okay so at least from the first glance I mean I'm speaking instead of Ivana here but on the first glance it sounds like it's a pretty early development stage there I don't know Ivana do you want to say something yeah from my view I think it's so early that they don't seem very open for interaction yet they have one general slack channel that it's not pretty active they don't have any meetings so issues opened from external people or with no comments and or pull requests are from the few people that started the project so I think I even had some ideas where to start from if we want some integration but uh I at this moment I'm still not sure how we can interact well with the SMI community I think we might need to wait a bit yeah I mean my my general council would be probably patients I've worked with tons and tons and tons of communities including lots of communities and their initial stages and starting a community is way harder than it looks um and so usually when I see a community that's like the one you just described for SMI that's literally less than a month out of the gate and just seems to be having trouble setting up for interaction you just check back in a month they usually much better yeah even the the initial network service match meeting started off like that like it was me and Ed and a couple people hopping on as like man we need to get more people on here so yeah I think it's an important problem though so and the people driving it are pretty high they do have some pretty high-profile people on it so I think they will if they if they keep up with it I think they will build a community around it yeah I still have concerns it seems like SMI will be successful at least from politics point of view they are going to push for it but yeah I think it's good to wait a bit I'm continuing with your research I may start trying some things but let's see how things are going to develop in that direction I find that East York people are not very interested yet but it's still not sure maybe they will become interested in the near future yep and in one of the things I mean one of the nice things with all these layer 7 service meshes you're dovetailing with them is lovely but but do the wonders of layer separation and networking you can run the layer 7 service mesh on top of network service mesh and they don't have to know and so that that you know and we don't have to really know that much either there are a couple places we could probably do some helpful things but yeah so I mean I think it's excellent to continue reaching out but I'm perfectly fine with whatever success we get and getting to respond I even think that the SMI people even if Istio contributors are not interested I think that SMI contributors should make some integration or even try to to run together with Istio to integrate it on their own in order to to be able to provide the common API and not have it separated having all the small meshes and excluding Istio I think they're going to do that. The two areas I would recommend looking at for an SMI integration at this point and not even implementing but even just like speccing out and and thinking about it but the first one is enablement being able to to add a service mesh by policy and so we we already have an Envoy example which can be used as a starting place for that and that that Envoy were happened to be configured to work with a service mesh then that would get you an initial interaction. The second one as well is around enabling federated communication and federated workloads so think of like right now a huge problem that many groups are trying to solve is like how do you get cluster to cluster networking to to work and many of them are unfortunately going down the path of let's assign a one block of IP addresses to this to these this cluster and a second block of IP addresses to these clusters that way they don't and that way they don't collide and then we'll turn them into a into a giant network with east-west traffic between them and rather than go through that perhaps a question would be how do you connect workloads to workloads between between clusters using network service mesh which could include an Envoy or service mesh component to that so those would be the two areas that I that I would recommend investigating and I'm sure there's there's others as well that that you may be able to think of. I have heard from some sources that the federation is not really a topic for the SMI I mean they want to keep this separated I mean I don't have any confirmation to that but that's what I've heard so that actually brings an interesting topic because I don't know we don't have this really on the road map and our actually our road map seems to be focused on very specific I don't know domains or points that we want to tackle and play with but do we want to put on the road map maybe in the distant future but still kind of at least starting the discussion with I don't know fellow CNCF projects like Lincardy or maybe not that fellow CNCF projects like these two some kind of underlying integration and I don't know verification and maybe maybe mesheries I don't know they all seem to be connected to all these technologies so what do people think about this I think generally speaking collaborating across various projects whether in the CNCF or not I know Istio is currently not a CNCF project yeah is is probably goodness I know for example I've been talking a lot with the spiffy spire guys and I don't think the spiffy spire guys are used to use network service mesh but they look exactly the right tools for solving a lot of problems that we have you know so there's there's a lot of good stuff going on there yeah spiffy is really spiffy I think I think that that's actually a really good example because when part of part of what we're going to have we're going to have to solve is how do you know who you're connecting to is actually who they say they are and not just some intercepted or random thing and spiffy inspire give us a a pass to that but I mean if we take our multi cluster I don't remember where it is here maybe on the on the roadmap so like floating in their domains or in general inter domain if we have this feature and we like enable like traditional service measure let's say level seven application service mesh on top of network service mesh then essentially for them it would be pretty transparent that there is kind of this inter domain thing going on because exactly from Istio's point of view it's just the one domain right yeah because they're running up of a network service map as network service of some kind and the fact that it happens to touch dozens of clusters is completely invisible I mean do do we think that that this also really huge problem that they're trying to solve in like other ways I know that council is doing some some some pretty smart things regarding this but I'm not sure how how the other guys are doing yeah if you if you step out of the service mesh domain for a bit then you'll start to see some other types of activity so that's that's why I mentioned like like the joining of L2 or L3 domains is is I know I know of a couple efforts where people our groups are trying to to perform that using various types of of the VPN strategies and so on and I can tell you they've worked on those for well over well over a year and still haven't produced anything that that's usable at this point and when I talk with people in those areas they're frustrated and so that inter domain problem is definitely a big problem and it's definitely something that that when we get around to solving it we can we can get a lot of mileage out of out of the inter domain because we'll we'll effectively enable the the federated multi-cluster and not just federated multi-cluster but across other other devices and domains as well but we'll we'll solve it as part of that as part of that effort and so I'm expecting to see not not a huge amount of stuff but for but for people or companies who really need it like they'll they'll definitely be a lot of excitement around it I think that that we mentioned the last time that essentially one of the steps that we need to take into the relation of how to say maturing as a project and moving into from sandboxing to incubation is to really have some some okay is as production as possible deployments I mean even if someone deploys NSM in a CI that would still maybe considered as some kind of production type of thing for some people but I mean I'm looking at the the technology here and I'm trying to to see what we'll what is the shortest path to to kind of attracting people start you know deploying it and using it yeah no totally I I think the inter domain is the one that's going to be the big the big man you know because inter domain and floating inter domain because that allows people to break free of their cluster right and that's huge you know and and we've got lots of other people who are interested in other things like the SP guys are going to be interested in harmonics and SRV and they're going to be interested in SRV six etc but I think the broader enterprise world like floating inter domain will blow their minds so I would say even on the service provider space right like I host like hundreds and probably someday thousands of clusters and sometimes it's workloads that span these clusters so even in the MSP space inter domain is everybody wants to do hybrid cloud right like multi cloud like it doesn't matter if you're a two location pizza shop that uses AWS or like you just want all your crap to talk and do that so I think that's the big win and then on the left security nothing else matters if you don't have security because nobody wants the first person that's on the news their service mesh like just put everybody's information out on yeah I mean intermain isn't even he's even actually really doable and because it means not isolation and so you've got to be able to to deal with that and that's where I had that's where the conversations were going on with um that that's the wrong link sorry I put the wrong link there yeah back to the spec board you can find the inter domain that was my screw up um but you know that that's why I spent so much time talking to the spiffy inspire guys um about security um during NSM connection request closed as the security won't be others interdomain yeah but the that's why I spent so much time talking to the spiffy inspire guys because it turns out the heart the the good news is the really hard problem for us for security the spiffy spire guys are solving which is how to handle authenticatable identity and then the um from there we have a little bit of problem around provenance that turns out to be relatively easily solvable and it's in the security spec um and then um you know from there there will eventually be an authorization policy problem but I think we can shove that off on the open policy agent guys for authorization um because the people are going to have all kinds of opinions about who uh who's authorized for what but if you can secure identity which is what spiffy inspire does for us and you can get the information about provenance in other words who's been involved in getting this from me did you and me then you're in pretty good shape so and so if folks would please go look at the interdomain spec again even if you already have it's been updated a bit in response to comments um and also go have a look at the security spec the more eyeballs for security stuff is always better and the other thing I would have mentioned briefly is I know that there was a gateway spec that Matthew had opened and I need to actually go and put some comments because I think the recent update to the interdomain spec um actually will cover the gateway thing but I would want to make sure that Matthew actually can curse there because he often thinks very deep thoughts yeah I'll revise my comment about uh limited um versus big versus huge I think the use case itself is massive I think the number of people who are going to be willing to implement it is going to be limited which is good for us because that means we've been pretty much work with them once we demonstrate something working and we'll get and that should be that should be an easy driver for us to to grow the enterprise community quite significantly um cool is there um is there any other comments on the uh on the technology tree and that's a it's a pretty fun area yeah everyone likes the future even it's being distant right oh yeah actually one of the things that I really love about the technology tree is it reminds me of um of playing a video game and it's like you can choose where to where you spend your skill points and um which which tree do you want to get first and you know and it's the type of game you can get all of them but you have to but that early game you know and you're how fast you can level up depends on where you spend those skill points so it's almost like you tell you're telling me that that I can't have all the things all at once well you you can we just need to spend well I don't know maybe in our domains the exploit that puts your guild five months ahead of everybody else yeah okay we have 10 minutes left is there any any pressing topic that we want to go through one last thing so the community uh the uh so the patch was merged for the um for the code of conduct so as of as of today we are following the code of conduct and again if anyone runs into any issues you can find us and on the owners.md and we have our private emails listed there and you can also read the the actual document from the code of conduct to see where the to see where the mediator is if if you feel that you're not able to get resolution from from us as the maintainers so thank you all for for respecting not only the document but ourselves as well and with that are there any any other last comments before we build back a few minutes of our time okay with that we will see you we will see you all on either tomorrow for the documents call or next week at the same time actually that's that's probably a thing because next week me and you are going to be in china I hope some of the other people here maybe so I don't know it tomorrow this meeting cover waps with another meeting so maybe we are going to join for half only so I guess what you're saying is because next week you guys are going to literally be in the other side of the world um that that you may find this meeting time difficult is that what you're saying Nicolai uh yep no that seems perfectly fair I will definitely be here and if need be can pick up the gauntlet it should be 11 p.m. so I'm hoping to make it if if my internet connection and allows me to so great well with that um I think we've come to the end of our agenda and we will see you all at them at the same time next week enjoy the rest of your day cheers thanks bye bye thank you bye bye


Our speaker today is Guillaume Lamper. He studied mathematics at the Ecole Polytechnique in Paris and then moved on to Carnegie Mellon to do a master's degree in natural language processing. He moved back to Paris to do his PhD, which he finished in 2019. And he is now a research scientist at Facebook AI Research. A lot of his work centers around machine translation in an unsupervised way. And he has been awarded with the best paper award for his work on this topic for one of the major conferences in the field. The topic of his talk today is not going to be language processing or machine translation per se, but rather what understanding this topic can teach us when it comes to manipulating symbols in mathematics. So on that note, please Guillaume, the stage is yours. Hi, thank you for inviting me. So I'm Guillaume Lamper. And today I will talk about deep learning for symbolic mathematics. It's some work that we recently published at the iClear conference. So the motivation behind this talk was that if you look at current results in deep learning with neural networks, is that neural networks they perform extremely well on a wide variety of tasks. For instance, in computer vision, whenever you have to classify images, when you want to segment objects in an image, or even if you want to recognize faces, if you want to recognize speech, or even in natural language processing, for instance, if you have problems like machine translation, so let's say translating automatically from let's say English to French, or if you want to classify whether a sentence is expressing like a positive or negative sentiment. For all these problems, whenever it involves classification, neural networks work really well on their main state of the art in most of this task where we're able to apply them. But so what we wanted to see is whether it's possible to use neural networks, but in mathematics. So people have tried on, but the problem is that there has been a very limited success in simulink computation with deep learning so far. So surprisingly, what we have is that neural networks, large neural networks, they can be pretty good at, let's say translating English to French, but this model we actually perform very poorly whenever they have to do some very simple mathematical arithmetic tasks, for instance, integer addition, or even if you ask a neural network to multiply two very large digits, let's say two numbers with five digits, it will actually fail, or it will actually succeed, but only like in 80% of the time. So it doesn't work really well. And the thing is that most people in the field of deep learning for symbolic math have actually focused on these very simple arithmetic tasks. The reasoning being that if neural network are not able to do addition and multiplication, then there is actually no hope that we can use neural networks to perform some complicated math, like solving theorems or computing integrals or solving differential equations. But the thing is that actually humans are also not very good at this. If you give somebody like two large digits, to the large number of five digits, it's certainly actually that somebody can actually compute the product of these numbers. Of course, actually people have to write it down. They cannot like mon-telecomput this very quickly. So, but still humans are actually pretty good at like solving theorems on like computing integrals. So maybe we have only tried so far to apply neural networks in math, but on some of the bad problems. Maybe neural networks can be used in mathematics, but maybe they will actually perform better in more complicated problems than addition and multiplication. So this is what we wanted to see in this paper is whether it's possible to apply neural networks on more elaborated tasks, knowing that they actually don't work well on simple tasks like multiplication. So we focused on two different problems in mathematics. So one is a function integration and one is solving differential equation. So the thing, the nice thing about these two problems is that they can be seen as a particular instantiation of a machine translation problem. But instead of let's say transforming like an English sentence to a French translation, what you will do here is basically map, let's say an expression, a function to its integral. So you basically have to translate like a problem into a solution. All neural networks perform really well in this task like sequence-to-sequence model. Sequence-to-sequence networks are very well-studied and they perform very well. So we can see maybe whether they can be used for these problems. And it's also one of the reasons why we had some hope that neural networks may work well in this problem is that it's also a problem that involves pattern recognition. For instance, if a human is trying to actually compute the integral of a function, usually what you try to do is to organize some specific patterns. For instance, if you see that there is like f prime times cosine ucf for a particular function f in the function that you are trying to integrate, then it's likely that sine ucf will actually be part of your primitive. So if f is a small function, that is easy, but if f is like a complex expression, it's gonna be more difficult to notice these patterns. So it's really like some pattern recognition problem on neural networks that actually exceed and they perform extremely well in this pattern recognition task. For instance, when you do image classification, what you do is that you try to detect some specific pattern images or if you want to classify sentences, you try to detect some particular subset of words. So here it's actually a little bit similar in this sense. The difference though is that neural networks operate on sequences as you connect, you cannot directly feel like an equation to a sequence to sequence model. And also one of the bigger requirement is that they require a lot of data. If you train a machine translation system with let's say thousands or thousands of thousands of, let's say English to French translation, then your model will actually perform poorly. This neural network, they work well, but the requirement for them to work well is that you need to train them on millions or like thousands of millions of examples. For instance, in English to French translation, we use like typically 100 million sentence pairs. So you have basically a data set of 100 million English sentence with like the 100 million corresponding French translations, which you don't have like in function integration and differential equation. So we have to address this. Okay, so the outline of the talk will be the following. So first we're gonna see how we can actually make the problem compatible with sequence to sequence models. So how we can actually use sequence to sequence model to do function integration or sort of differential equations. Then we are gonna see how we can, I mean, what are the models that we use and also how can we create data sets because there does not exist any data sets of like thousands of thousands or millions of integrals on differential equations. So we have to like regenerate them so we can train the model on them. Then we are gonna see what is the performance of the models, the neural networks train on these data sets and how they compare with like other solutions, like commercial software, like MATLAB or Mathematica, that have been used for many years and that are like well established and developed. And finally, we're gonna discuss something which is important in machine learning, which is in machine learning, which is generalization and which is basically the question of if we train a model in a particular data set, do we have the guarantee that it will also work well on another data set, which is not guaranteed in machine learning. Okay, so the first problem is basically the question, how can we use mathematics as a natural language? Because basically sequence to sequence models so far have been mainly used to translate sentences or to generate text. But here we work with not text but with mathematical expression. So the way we do this is that we actually have to transform equations into a list of tokens. So a token can be basically, for instance, we have one token for Senus, we have one token for Cosinus, one for Tangent, we basically have one token for every possible function that we want to use. We have one token for X, one token for Pi. And the idea is that our neural network model will have one embedding. So an embedding is essentially a vector. So we will have one vector for each of these tokens. And if we have a large number, so let's say for instance just a number like 1,414, the question is how do we represent, how do we give this number as input to a neural network? So here what we do is that we split this into individual digits. So here we're gonna represent this as a sequence of five tokens. It's gonna be like plus one to one four. And the idea is that if there is, I mean since there is like an unlimited number of integers, you cannot have an unlimited number of, you cannot have one embedding per integer. If there are like one billion numbers, you don't want to have one billion vectors. So you split these numbers into digits. So you can actually represent any number using a sequence of tokens. Yeah, and the other thing is that we do, I mean the first thing that we actually do is that we take an expression and we transform it into a tree. And then from a tree, we can actually transform it into a sequence.  We just take an expression and here with what we can do is just transform this into a binary on unary tree. So here for instance, if we have two plus three times five plus two, then the main node is gonna be a plus and we are gonna add two to basically the multiplication of three with the addition of five plus two. So this expression is actually equivalent to this tree here. And if we have this expression here, we can do something very similar. The difference here is that we have also a co-sinus. So here this is not a binary but a unary operator. So we have a co-sinus here that will actually take as input as a multiplication of two with six. And if we have some more complicated expressions, let's say you have some, let's say you want to work with differential equations. So you want to represent the derivative, the differential operator in your input. So what we can do here is have a tree. And for the value, for the root of some trees, what you can have is the operator differential. So here, if we want to represent this part of the expression, we basically have a tree of depth two. For the root node is differential operator. And what we differentiate is what is on the left and what is on the right is the variable with respect to which we differentiate what is on the left. So basically this expression here can be represented as this tree. So the first thing that we do is that we transform our expression into trees. And once we have a tree, what we can do is simply enumerate the tree. So we use that prefix expression. So we simply transform these trees into prefix sequences. So for instance, if we have this expression here, we can actually enumerate the root node. We enumerate the tree starting from the root node. So we have plus, we have what is on the left, it's a two. And then we enumerate recursively what is on the right. So it's gonna be like times, then three, then plus, then five, then two. So we start from the root, then what is on the left, then what is on the right. And then we can actually enumerate this tree into a sequence like this. So to summarize, we basically take an expression, transform it into a tree. And then from this tree, we can actually transform this tree into a sequence. And all the time, there is like a one-to-one mapping between an expression, the C corresponding tree, and then the corresponding prefix sequences. And since we work with sequence to sequence model, what we are gonna do is basically give as input to our models this sequence of tokens. And we are going to train our models to predict another sequence of tokens, which is the case of function integration. For instance, the input will be like a function and the output will be like the token of the integral. So yeah, this is gonna be like the input and output of our models. So now, as I was saying, so if we want this model to perform well, we need to have very large data sets. And unfortunately, that does not exist very large data sets of function with their integrals or differential equations with a solution. So we need to find a way to generate these data sets. So for integrals, we propose three different ways to do this. So one is very simple. It's what we call it's a forward approach. So the idea is that what you can do is generate a random function F and then you can use some existing framework like SimPy or Mathematica to compute its primitive big F. And then what you can do is simply add F big F to the training set. So you add like a training example pair. And then what you're gonna do is feed to the model as small F and we are going to train the model to predict the sequence of the elements that represent the function big F. So it's like the most simple approach to do this to create a data set like this. The only problem is that, I mean, there are many problems is that first of all, you need to require on some external framework. So you need to use Mathematica or SimPy or any other framework that can do integration. And so because of that, it's unlikely that you will be able to perform better than these frameworks if you train in your network on the data that they generate. The problem is that it's also very slow. For instance, there are functions for which Mathematica will take like thousands of seconds to compute the primitive on some time it will actually just fail. For instance, there are many functions for which there exists a closed form solution if you want to compute their integral but just Mathematica will not be able to find it. So again, you will only generate a subset you will only generate a data set which is composed of the subset of functions that Mathematica is able to integrate. So you will not generate like a perfect data set. So here are some examples of functions that you are going to generate with this approach. So here you basically have what is generated on the left. So here we randomly generated these functions. And then we ask Mathematica to compute the integral and we found these functions. So here what you can see is that the input is going to be very small and the output is very large. So here we train the model to predict what is on the right given what is on the left. So it's kind of, again, it's some sort of machine concentration problem where the input is this and the output is that. And yeah, and what you can notice here again is that the output is much longer than the input which is not always the case on your, that's what we're going to see with the other data sets. So the second approach to generate example of function with their integrals was to use what we call a backward approach. So here the difference is that we do not start from the function to integrate but we start from the integral itself. So we generate to random function but we call it big F. On here we can simply differentiate it to get the derivative small f. And then we just add again F on big F to the training set. So the advantage of this approach is that it's extremely fast. And it never fails. When you have some function, you can always compute it's a derivative and I mean assuming it's differentiable. And the problem though is that you will tend to generate functions that are extremely long. Like the derivative are going to be very, very long. Typically here are some examples of generated sample from this data set. So here we generated what is on the right. So we generated small functions. And if you compute the derivative of this function you will get what is on the left. So here you can see that the inputs. So what we are going to give to the model is composed of is basically very long functions on the output is very small. So you have very small solutions on the very long problems. But okay, on the other problem was that if you take an expression like for instance this one that we had before like x power three times log of x square power four. So the integral of this function is this very long function here. So the thing is that if you generate a function around them Lee, it's absolutely and I mean it will never happen in practice that you generate this function around them Lee. I mean the probability that you generate this exact function is almost zero. So basically with the backward approach we'll never be able to add some example pairs like this in your training set. You will only have a function with a very small output on the very long input. You will never have something like this because the probability of generating a function like this is basically zero. So this is why we have a third approach that we call basically the integration by part generator. And the idea here is that we can generate two random functions. So big F on big G. We can compute their derivatives.   I mean we know these functions so we know the product. And if we have, if we know the primitive of this because we have already computed it and it's already in our training set then we can actually automatically infer the value of big F times small G. On this verse, if we know the integral of big F times small G, we can infer the primitive of F times big G. So the advantage of this method is that it generates a function that you will not generate with the other generators. But the problem is that it is very slow. But the nice thing with the last two approaches, the backward approach on this approach is that you can actually generate very large datasets without relying on any external frameworks like Mathematica or Simpa. You can actually generate that using like very simple code on no external API framework. So here are some examples of functions on primitives that you will generate with this approach. So here we generated basically these pairs where the input is again on the left and the output is on the right. So we are gonna train our models to generate what is on the right given what is on the left. And now you can see that examples like this like X power three times C new C public of X that has this particular primitive. We know of this in our training set while the backward approach will never have given this. We will never obtain this example with the backward approach because typically this requires to do multiple integration by parts. And this is why the output is very long. Okay, so now we work with integral but we also wanted to see if our model is able to solve differential equations. So here we have the same problem. We need to generate a dataset of functions of differential equation with their solution. So what we could do here is generate a random differential equation and try to solve it. But the problem is that if you generate a random differential equation you have no guarantees that there exists a solution to it or especially not in a closed form solution something you could express with symbols. So we do something a little bit similar to the backward approach which is that we generate a random function and then we try to see what is the differential equation satisfied by this function. So if we have first order differential equation we know that the solution, I mean there are many solutions but they are basically all the same up to a constant C. So what we do is that we generate a random function. So to generate a random function what we do is that we generate a random tree and when we populate the leaves of the tree we put some X on one constant C. So just one constant C. And what we do here is that we express the constant C in terms of everything else. So in terms of f of X. So to do this, so we basically solve in C. So it means that, I mean here it's pretty simple. You can just move the X on the left hand side. You can exponentiate this to remove the log. And if you put the X on the other side you have that C, the constant C is equal to X times exponential of f divided by X. Which means that this function is constant of X. So if you, because it's equal to a constant. So if you differentiate it with respect to X it's equal to zero. So you have that this expression here is equal to zero. And now you can actually simplify this a little bit because you know that exponential of f divided by X will always be non-zero.  X y prime minus y plus six equals zero. And then you can verify that this function is actually a solution of this particular differential equation. So overall you can actually generate random functions with one constant C.  And you can also do this at the second order. So here is another example for the second order with again a simple input. So here we have not one but two constants, C one on C two. So we generate a random function with C one on C two. Again we are going to express one of the constants. So C two here in terms of everything else. So f X on C one. So we have that C two is equal to f of X exponential of X minus C one exponential of two X. So this function here is a constant of X because it's equal to a constant again. So we can actually differentiate this with respect to X, it will actually be equal to zero. So we have that this big expression here is equal to zero. And similarly here what you can do is again express C one because here you have C one there. We can express C one in terms of everything else. So you have that C one is equal to this expression here. And now if we differentiate this with respect to X we have this following formula. So exponential of minus X divided by two times f prime prime minus f. And if we simplify this by this factor which is always known as zero we get this following differential equation. And now we can actually see that this differential equation has indeed this particular function as solutions. So yeah, so here what we can do is generate like a 10 millions random function with two constants do this process.  And we can do this again without using any mathematical framework. Okay, so for the data set in practice what we do is that so for all of this process we start from by generating a random function on to generate a random expression we do this by generating a random tree. So we generate random trees with up to like 15 internal nodes. And for all this node we are going to populate them either with unary and binary operator. So for the binary operator we have the addition like subtraction, multiplication, division and we have all this function for the unary operator so exponential log square root the trigonometric function and the inverses on the hyperbolic functions. And yeah, so now we can actually look at how many expressions exist we can generate. So here you have different curves. The X-axis is the number of internal nodes of the tree on the Y-axis is the number of expression and is the number of trees associated. And L is the number of possible leaves of the tree. P1 is the number of possible unary nodes. And P2 is the number of possible binary nodes. And you can see that for instance if L is equal to one and P2 is equal to one you have that this is a number of binary trees. So this is like the number of binary trees like the Catalan number that you can generate for this different number of nodes. And you can see that if you also allow your model to generate unary trees you have more possibilities to do this. And now if you allow yourself to have like 11 possible leaves, 15 unary nodes on the four binary nodes you have the blue curve which is basically what is, I mean, which is in our case this is the setting that we use. And if we have 15 internal nodes we basically have like something like 10 power like 45 possible expressions. So we have basically an almost unlimited number of expressions that we can generate to generate data sets. Yeah, so here just very quickly so this is a number of examples that we generate for all of our data sets. So for the forward, backward and integration by parts generators we generate 20, 40 and 20 million examples. And for the differential equation we also generate 40 million examples of input and outputs. And yeah, so sometimes we can have we sometimes have very long expression with up to like 500 tokens or like 500 tokens for the input of our differential equation. So as the examples that I show here are only very short but the model is typically trained with extremely long examples. Okay, so now in terms of results on how it compare with other softwares. So we compare the model with Mathematica, MATLAB and MAPL. So for the integration we only compare on the backward generator. I mean comparing on the forward generator will be possible but it will be not very interesting because Mathematica will get 100% on this because by definition the forward generator is the function that Mathematica is able to integrate. So we evaluate the performance of the model on the backward generator. And here we can see that Mathematica has like 84% accuracy while our model is basically almost at 100% accuracy. So the way we measure the accuracy for the model is that so here we use like a sequence-to-sequence model. So the way a sequence-to-sequence model works is that it takes as input a function and it will propose several candidates for the integral. So it will say, okay, here is the top 10 candidates that I think is a solution of this differential equation or that I think is an integral. And what we can do then is just, let's say we're doing integration, we can take like one output of the model, we can just differentiate it and if we retrieve the input, if we retrieve the function that we wanted to integrate, we can see that this was actually a valid primitive, a valid integral of the input function. So here you have like three results with beam size one, beam size 10 on 50. So beam size 50 means that we actually let the model generate 50 solutions on the model we actually get a point. So it will score one point if one of the 50 solutions is actually valid. On beam size one, you just let the model generate one answer and you consider that the answer is correct. I mean, if it's incorrect, you basically don't let the model generate another solution. And okay, and you can see that here the model has the most 100% accuracy. And in differential equation, you can see that mathematics has like 77% accuracy for the first order, 61% for the second order. While the model will have, if you allow it to let's say make 10 guesses to propose 10 solutions, we'll have like 94 or 73% accuracy. So for differential equations, what we do basically is that we give as input to the model one differential equation. We ask the model to propose one solution and what you can do is just take this solution, plug it into the input differential, I mean differential equation. And if it's equal to zero, it means that this was actually a valid solution. So we can actually easily verify whether the model outputs are correct or not. So we still need some symbolic framework to verify the generations of the model. But the idea here is that the model is very good at proposing candidates on solution. And then you can actually verify them using a symbolic framework. Okay, yeah, just one other thing. So here we actually use a time out of 30 seconds for Mathematica, which means that if the Mathematica doesn't find the solution in 30 seconds, we actually consider that it fails. But in practice, we try to put 10 minutes, but this doesn't really change the results. Usually Mathematica either returns the solution instantly or just doesn't find the solution at all. So here are some example of functions that Mathematica is not able to solve, but that our model is actually able to solve, for which our model can find the solution. So for instance, the first example here is a function to integrate. So you have to compute the integral of this. So Mathematica is not able to compute the primitive, but here the model is actually able to find the solution almost instantly. So here it's basically the arxinus of this function here. So interestingly what happens here is that if you give to Mathematica, not this, but if you slightly re-factorize the denominator, so if you put one minus basically this square, so Mathematica recognizes that it's actually the form of the inverse series basically. So if you slightly rewrite this by one minus the square of this, then actually Mathematica can find the solution, but if you expand the denominator like the way it is right now, Mathematica just fails to compute the solution. And here you have an example of a first order differential equation that our model finds and that Mathematica cannot find. And here you have the same example for like a second order differential equation. So in this example, the model instantly finds the solution while Mathematica fails completely. So here are some other examples of generation, which I think are quite interesting. So here you have some inputs, which is a first order differential equation, so this, and the model, as I was saying, can propose different solutions. So if we have a beam size of 10, it means that the model is actually able to propose 10 different solutions. So here the model will suggest 10 different outputs. And whenever the model proposes a solution, it also proposes a score. So here, for instance, the score is typically like a log probability. So if you have a score of zero, it means that the model was confident with probability one, that this was like the good hypothesis, the good solution. And interestingly here, what you can see is that all these 10 different solutions proposed by the model are actually all equal. All these 10 different equations are actually exactly the same, except the last one. But if you actually take a different value for the constant thing, you actually see that it's actually exactly equal to this nine other expressions. So interestingly, the model here was able to find that these 10 different formula are actually all equal, even though we actually never trained the model to do this. So we only trained it to compute the integral of functions, but the model was able to find on its own that these 10 different functions are actually all the same. Yes, so earlier in the talk, I mentioned that there were some differences with the generators that we can use for the forward, the backward on the integration by part, the difference being essentially that the expressions are very different lengths. Typically, the forward approach tends to generate functions with a small number of tokens, but a very long number, sorry, a small number of tokens in the input, so small input functions, but with a large number of tokens in the output. So the integrals are very long, like the green curve here. The backward approach was exactly the opposite. So we have very small inputs, sorry, very small integrals, but the outputs tend to be actually very long sometimes. On the integration by part method was actually a bit in between. So we have, at the same time, short inputs and short outputs. So in machine learning, what happens is that if you train one model on a particular distribution and if you test it on another distribution, which is very different, it will tend to perform poorly. So this is what we wanted also to investigate here. So what we can see on this slide is that, so we train four different models. We train one model on the forward dataset, one on the backward dataset, one on the backward plus the integration by part dataset and one on the three dataset that we can get in idea. And for each of these four models, we basically evaluate the performance on the three datasets. So what we can see is that if we evaluate on the forward dataset, we perform very well on the forward dataset in blue here, but we tend to perform very poorly on the backward approach, like functions to integrate that come from the backward generator. We don't even have to 20% accuracy on this. On the backward approach is similar. If you train your model on the backward dataset, it will perform very well on the backward approach with almost 100% accuracy, but you will get only 20% accuracy on the forward dataset. So now what we can do is concatenate the backward data with the integration by part data. And what is nice here is that these two datasets were generated without using a Mathematica or any symbolic framework, just like some very simple Python code. And if you train a dataset on this, a model on this dataset, you will get so like an almost perfect accuracy on these datasets, but you will also get more than 50% accuracy on the data coming from the forward dataset, which is basically the dataset on which Mathematica is able to compute functions, integrals. And finally, if you train a model on everything, so on the forward dataset, the backward on the integration by part dataset, we can see that the model get an almost perfect accuracy, almost 100% everywhere, which means that if you know exactly the distribution of the function that you want to integrate, if you have a generator for this distribution and you train a model on this, then your model will perform very well on this distribution. But the only issue here is that we can never be sure that that does not exist, and that probably exists. Another distribution different than the three datasets where the model will actually perform very poorly. Maybe there are expressions that are not generated by the forward, the backward on the integration by part approach, that this model will not be able to integrate either, which is basically one of the main issues with machine learning. Yeah, so finally just some other results on generalization. So the forward model, so again, was trained on the function that the SIEMPI was able to integrate. It's basically, so our forward dataset was generated with SIEMPI, which is like the Python equivalent for Mathematica, even though not as performant. And we only train our model on the subset of functions for which SIEMPI is able to compute the solution. And interestingly, at this time, what we found is that this model was actually able to integrate function from the backward dataset that Mathematica is not able to integrate. So that SIEMPI is not able to integrate. So for instance, if you look at these inputs with these particular solutions here, these functions, our model is able to integrate them, even though SIEMPI is not, which is kind of surprising because in the end, the model was only trained on the subset of functions that SIEMPI is able to integrate, which means that in some ways the model was able to integrate, to learn, to generalize beyond the distribution of SIEMPI. And okay, so that's it for the talk. So the paper is online, as well as the code to reproduce the results on the pre-trained models. And I'm happy to take questions now. Wow, cool. Thanks a lot. That was a great talk. So questions. I would encourage people if they have a question either to raise their hand or in case they think their mic doesn't work, then please type them in the chat and I will read it out for them. And I can see we have one question there. Louis, please go ahead. I wanted to ask a question about generalization.  And I guess you just have to differentiate the answer yourself and see if it works. Is that it? Yeah, so basically, if we are given a new function, so the thing that our model doesn't know, our model can propose solutions, but it will never know for sure whether this solution it proposes are correct or not. So the only way to do this is to actually, yeah, we take a function, get a solution, differentiate it to actually see whether we retrieve the input function. And if we do not retrieve the function to integrate, it means it was not a correct solution. So we can actually take another one, I mean try another one, but yeah. So this model here is basically used not to replace like formal system, but just be used I think on top of formal systems. So you actually propose this candidate solution and then a formal system can verify whether they are correct or not. You have no other choice. The whole thing looked pretty impressive, but just one other question, and that was that you have this example where the sort of 10 best solutions, the best 10 solutions are all equivalent to each other and they have different scores. I guess the network doesn't know that they're equivalent to each other, is that right? Yeah, exactly. So basically what happens is that the model is generating one embedding. So like for each of this, whenever you feed an expression like this as input, your model would generate one vector. So it's basically one vector of dimension 500 and it will actually decode the solution from this. And here it will actually decode all of them, but it would actually, yeah, it means that basically there exists like a space of dimension 500 in which all of this expression, they have a very similar vector, like they're close in the vector space. But the model, I mean the model can suspect but it will never know for sure that they are correct. But in practice, it's kind of what we observed. If you have that two function have a very similar embedding representation, even then, even if they are written very differently, it means that they are basically equal. But yeah, we can never know for sure without like a formal system on top of this. Yeah, anyway, looks very impressive. Good talk. Thank you. Okay, there's also a question on the chat by Peter who asks if you have considered the structure of the algorithms that are used by these symbolic packages like Sympy or Mathematica and whether you have considered what first-year mathematicians are learning and whether or not you think this could help to find better models that can do the same job? Yeah, so for Sympy, we didn't really dig too much into it.  What we tried to do was to see whether we can understand what the model is doing by for instance, looking at the attention heads and basically, it's usually, yeah, the problem with this model is that it's usually a bit hard to understand why they output this or that. They basically don't tell you why they actually output a particular thing or another. But there are ways to basically investigate this. For instance, the model can actually predict a token and you can look at what it's looking at to predict a token. For instance, maybe the model will actually output, start output in Cosinus, only look at what it's looking at in the input before it predicts Cosinus. But the problem is that when we did this, so this method they were developed to analyze the model in natural language processing. So for instance, when you want to understand why a model is predicting this particular label for a sentence. So we tried to apply these same techniques to understand what is happening in the model. I mean, why it's doing this or this or that, but we didn't really, it's very difficult to interpret. We were not able to find like any new algorithm or something like this. Even though, I mean, the model is definitely doing something, but we are just not able to understand what it's exactly doing. That's really, I mean, we will need to actually investigate this a lot more to understand how it's working. Amen, please. Yeah, thank you. Very nice talk indeed. I would like to connect to Louis's second part of the question. Have you considered training the model on simple, equivalently reformed terms? So we see here these, the selection of terms which are equivalent to each other, but the model doesn't know it. But that's because the model hasn't been trained to know this. So, and this is a very cheap training. There is, it's very inexpensive to reshape expressions in quasi infinitely different ways. So if you were to do that, would there be some hope that the weights that we see here would actually become much more similar? And the model, in that sense, would start to know that these equations are the same. Yeah, that's a, yeah, it's a good idea. I think if we were to do this, we could actually train the model to do two things. So one is, I mean, the tasks that we have done here. So solving a differential equation on the second will actually be two. Yeah, we could actually train the model to map an expression to any other expression equivalent. Or we could actually do something like train the model to basically match, I mean, what you could do is, let's say we have an embedding for this formula and we have another embedding for this formula. So two vectors, we could actually train the model to minimize the distance between these two vectors. We could say, we could add to the loss, like minimize the distance between these two vectors. And actually, I think it would work. This element on its own is already, would already be an ingredient that would help existing packages like Mathematica to improve the integration ability. Because as you, have you just seen, you've actually put up an example of that same problem that Mathematica has, is able to integrate the same function if only it is written in a differently factorized way. So if there is an element that can simply re-represent elements of functions in different ways, then using those in conjunction with your standard symbolic packages can already be a big step forward. I think so, yeah. I mean, there are definitely some ways to get some gains into Mathematica. I mean, I'm not sure exactly what people would be interested in here. But for instance, here, the model was not working. I mean, Mathematica didn't work because the denominator here was not factorized. And having a model that actually detects this expanded expression on how they can be factorized, it's actually very simple to do for a neural network. And actually, yeah, it would be, I think, possible to take a model that has input functions, normalize them by either factorizing them or rewriting them in a way that are easier to understand for the above model like Mathematica. And yeah, I think these tools, like Mathematica and neural network would work well together. I think it's definitely something that will happen at some point. Only issue is that the neural networks are a bit expensive, typically. I mean, it's better to have a graphic card or it's a bit slower. It's not that bad, but yeah. At some point, I mean, it's obvious to me that in the future, these tools will be integrated in Mathematica, I think, because yeah, they will always be able to do things that we can actually not write with, like, I mean, we had a look at the SIMPI code very quickly for integration. It was really, really complicated. And it's obvious that a model that you can actually train like a black box like this on that performs so well should actually be useful for all these problems, I think. Thank you. Yep. Okay, Will, your question, please. Father, I take it you train the numbers on like decimal symbolism, right? Yeah. Okay, so do you have like any idea of like the limits you can take the model's understanding of the decimal number system too? I mean, even in like simple algebra, is there a limit up to which that it can either multiply or divide? Yeah, so yeah, so I very quickly talked about this at the very beginning. So we had initially like one way to represent this. I think, yeah, in the end, in the problem we used, we didn't have any float numbers. I think we implemented that at the very beginning. I think we, there were other things we wanted to try, like for instance, see from like some approximated solution of like an expression that can be written that can be written in closed forms. The model will be able to actually detect the closed form. So for instance, if you have like, let's say some approximated function, can the model look at this floating numbers and detect that, yeah, this is definitely looking like Cousinus of this and can actually map this floating representation to some, to the associated like closed form formula. But so yeah, that's something we wanted to try, but we didn't really investigate that. So I'm not sure, I'm not sure how it will work with the floating numbers because in the end, we didn't end up using them for the integrals. I'm not sure, sorry. I think it also, yeah. Oh, that's a perfect answer. Do you have any idea like whether this is good at just kind of like reordering? Have you tried it on more abstract reordering where you kind of just entirely randomly make up the problem that you have a set of rules in which the symbols can be reordered and then train it on that? So what we tried to do, so I don't know if it's exactly related to what you have in mind. So what we tried to do is for instance, if we have expressions like let's say A plus B where the model is able to integrate A and the model is able to integrate B. So automatically it should be able to integrate A plus B by just summing the integrals, but somehow it happens that the model is able to integrate A and A plus B, but not B plus A. It happens, it's not very common, but we found some examples like this where just some things that should be automatically doable for the model are actually not. And we actually found a few instances like this where the model is actually not robust to this very simple modification. If you just, yeah, A plus B is working, but not B plus A somehow, but yeah. Do you have any ideas to like use like a GAN architecture in order to have one model producing more difficult problems and another model solving them at the same time or do you not know if that would work? Yeah, I think it's a good idea. I think only difficulty here will be that, I mean, as far as I know, we still don't really know how to use GAN for sequences yet. I mean, people have tried that for quite a while, I feel, I even spent quite a bit of time at quite a bit of time on this like a few years ago, but the thing is that when you basically want to differentiate, let's say your GAN generates something, that's an expression, let's say F on the model is not able to solve this, then you want to give this feedback to the model, but you don't know exactly how to do this. I mean, there is no like a continuous signal you can back propagate through this discrete generation of tokens, so it's like discrete output. So you cannot differentiate with respect to that. And because of this, we don't really know how to do it. GANs are working well in images because like when you generate an image, basically an image is like a matrix of float numbers between zero and one. So we know how to do this well and the signal is going through these images pretty well, but we don't know how to back signals through the tokens. So I mean, it would be, if we had a way to do this, I think it should work, but we don't know how to do this yet in deep learning. Okay, thank you very much. Can I ask what the structure of your network is? Yeah, sorry I didn't talk too much about this. So it's basically just a transformer model. So there's this paper, like the attention is all you need, like that presented the transformer architecture like in a few years ago. And we basically use like the vanilla implementation of this model. So it's like a six layers model with dimension 500. So it's a pretty small model. And something interesting that we found is that actually the model performs pretty well even with a very small number of parameters, like in natural language processing, usually for the model to work well, you want to have like 10 layers or dimension 1000. But in our case, we found it was just like six layers or even like four layers of dimension like 100. It's already quick enough to have like a pretty good performance, which also means that, which is nice because if you wanted to use this maybe like in some symbolic framework like Mathematica, you will actually be able to put a very small but still very powerful model. But yeah, we just used like the vanilla sequence to sequence model of the initial paper. There's another question on the chat from Herschel who is asking about your compute requirements. So how many CPU or GPU hours did it actually take to get to that performance? So it's very fast actually. You can just try and like a model in a few hours, even in one GPU. I mean, I suspect if you train it on your laptop with your GPU, it will also work after like one or two days, which is also, yeah, again, one of the nice thing with this dataset is that you can train on it very quickly unlike in natural language processing where you really want to have like eight GPUs. Here, we tried to scale the model to train on like 32 or 64 GPU, but the performance was basically the same as with just one GPU. So you can train this with a very small architecture on the hardware requirement, yeah. Only thing that was a bit expensive to generate was the dataset. For the backward method, you can generate the entire like 40 million example like very, very quickly, it's super fast.  So it's actually pretty long. But yeah, but the backward integration by parts are much faster than the others. So I think the limitation is that the difficult part is to generate the dataset. That's, the rest is very inexpensive. But we released the dataset if people want to try running models on this. Okay, thanks. So I see no more questions. So maybe ask one last one myself, which is, which I apologize for it because it's very open in the question perhaps, but do we have any intuition for how far this approach can be stretched? So in other words, is it conceivable that you have a model that you train on all the two statements that you generate starting from some axioms and then you expect it to have some intuition for if you give it a certain theorem whether or not that's, that has a proof in that axiom system. Yeah, it's a, you mean like theorem proving basically? Yeah, it's like, yeah, it's, yeah. So it's exactly what we are working on right now. We are considering like several systems like a light on the metamat. It's a bit more difficult to work with than this because so, so yeah, here what we do is that we basically have an input and we generate like one solution and then we, that is just like one step. You have an input, generate one candidate solution and it's done, you can, it's either correct or not. For theorem proving, yeah, you have this, you have to do this kind of MCTS research where you have like one goal, you apply a tactic, you get like some sub goals, you try to serve the sub goals and you do this recursively. So you very quickly have some gigantic tree and it's a bit more challenging and much more difficult than this problem. Also the systems that have been developed for, I mean the formal systems have been developed, I think, without machine learning in mind at all. So it's very difficult to plug some model on top of like a clock or light, typically. So yeah, but that's still work in progress. So don't have so much to say currently, but yeah. Great. So any last words, any last minute questions? If not, then I would like to thank you very much again, Guillaume. I've got one more question, sorry. Sorry, we'll go ahead. In terms of that theorem proving, are you looking at more kind of abstract algebra terms? Are you looking at so-called harder problems? I don't really know how to describe it very well, but yeah. Yeah, so we use, I don't know, so for instance in a light, you have like the proof of the Kepler conjecture which is there. So it's like a lot of theorems, a lot of statements, a lot of definitions. So we just use that and we just, I mean, we have like 40,000 proofs, formal proofs that we actually split into like sub proofs and we actually like, we have like 40,000 trees and we take a small subset of that to make a test set and see it's like what we use to work. Yeah, the issue that it's very difficult to generate large data sets. I mean, these techniques of forward, backward, you cannot do that in a light. But the math, I mean, the mathematics it covers is pretty wide. I mean, I think there is pretty much, I mean, there is not everything but there is like complex analysis, group theory, a lot of stuff, arithmetic. So yeah, we basically just, I mean, somehow we don't, I mean, we work with this data but we don't really understand the data ourselves. We just give it to the model and it figures what to do with it. It's a bit more, it's a bit difficult to interpret. Sometimes the model is suggesting some tactics but you have no idea whether it's a good idea or not to just try it on the, yeah, we are not familiar with the formal systems really. So yeah. Okay. Just one more. I don't know if you use like a kind of word to vex kind of pre-processing for your sequences but if you do, do you find that there's any kind of underlying manifold structure in the kind of the clustering of objects or symbols? So we don't use word to vex but yeah, there is some sort of manifold. It's basically like a bit related to these equivalent solutions here. So here we have 10 different solutions. And if you look at their embeddings, the location of their embedding in the vector space, they're actually very close to one another. And similarly, if you have something like, I don't know, if you have like procedures of 2x you'll have an embedding for that. And if you have the other formula which is equal to that, like 2x square of x minus one actually have an embedding which is also very close to that. So we observe that similar functions have an embedding which is close in the space. But yeah, so we observe it. Word to vex could actually be more at the token level. And here if we actually plot the embeddings at the token level, I'm probably, I think we will have that the sinews and cosinews are nearby but since there are not so many tokens, I don't know if there is so much to get out of that. It's interesting for word embeddings because you have like a, if you work with English you have thousands of thousands of words so you can observe things. But in our case, I don't know if, I mean, yeah, at the token level it's interesting, yeah. Okay, thank you very much. Thanks. Okay, there's one last question by Yangui who is asking a question in connection to a recent effort or collaboration by Google which they call the Ramanujan machine where they try to generate new identities, continued fractions and so forth. And where you had any idea of how this connects to your work and whether you could use models or your approaches like yours to generate new identities? Yeah, so I think I'm not entirely sure but I believe the Ramanujan approach was more like some, wasn't it some like more brute force kind of approach? I forgot if this was using some machine learning or not. But yeah, no, I'm not, I actually don't really know the paper very well so I don't have like an interesting answer for this story. But yeah, if you wanted to find some identity, I mean, I guess what you could do with these approaches, I mean, yeah, there are things you could try, for instance, maybe, I mean, you could generate like a random formula, like, I don't know, let's say exponential of square root of N divided by N plus one or something like this and evaluate this for N equal like a bunch of different values and then you could actually train a model to retrieve the original formula given the evaluation of this formula at different points. For instance, if you had something like, I mean, I think like there was a story with Ramanujan and when I think we knew the evaluation of the number of partitions of an integer at different points but we didn't know what was the exact formula for this and then you can have some intuition about the formula and just write it down immediately like you did without having a proof for that. I think you could maybe try something similar with the network by just generating random formulas, you can do this for a limited number of equations, evaluating them at some random points and then see if the model can retrieve them. I think yeah, it's something we slightly considered but didn't really have time to investigate much more. Okay, now that was it, wonderful. Thanks so much again, Guillaume. That was really great and thanks a lot to everybody who has asked questions and that's it for today. See you next week, bye. Thanks, thank you very much. Thanks. Bye everybody and thanks for the talk. Thank you.