Good day friends we'll get started in about a minute. Hi Lance. Hey Bjorn, how are you? Oh I'm good. Good. Finally back from a holiday. Oh good, how was it? Oh wow, complicated. That's the shot version. Ah holidays can be that way, can't they? Yeah. What's the, I've heard the old saying that no one needs the vacation more than someone who's just been on vacation. Yeah that's fitting my case. All right good stuff. Well I don't know where my other Ruth's ID friends are, hopefully they're coming, but if not, ah look at that. See you just have to invoke the friend. Hey Roto. Good morning, how are you doing? All right. All right, let me share my screen. Hopefully the right screen. Oh, hold on. Too many tabs in that screen. Let's make it simpler here. Okay, trying again. Okay, welcome. Well let's see. Let me give everybody the link first. Okay, there is the link for our meeting today. Welcome everybody to the March 27th, 2023. Aries Didcom v2 working group. I am your host Lance Bird from Ruth's ID and I have to remind everybody about the antitrust policy for hyperledger as well as the hyperledger code of conduct and feel free to add yourself to the attendees list if you like and is there anybody who would like to introduce themselves today? Okay, fair enough. Okay so today we want to cover IIW which is in April and we're doing a big Didcom Connectathon. I guess we're starting to refer to it as and that'll probably take up a bunch of our time but we could revisit AIP3 as well as some other things that we haven't gotten to since we've been busy and we can also anything else that you all think that we should add to our agenda for today. Happy to discuss. Okay, there enough. So in terms of updates I did not attend the Aries working group last week. Just super busy. So if anybody wants to give any updates for many of the Aries working groups or even things outside of Aries diff, trust over IP, other things like that, feel free. Okay, that's fine as well. Certainly the trust over IP trust spanning protocol stuff is well I guess winding down in terms of the exploration portion of it and so hopefully now they'll be kind of moving towards a consolidation point where they different proposals start to converge and define what overlaps and what qualifies as a trust spanning protocol and things like that. So Didcom v2 of course is front and center on that so that's exciting. Yeah, then we'll be ongoing. All right, so IW is in let's see how many more meetings do we have until then? We have today and then third and 10th probably the 17th will be canceled as a bunch of us will be flying or not necessarily canceled but maybe somebody else will have to lead it. So we essentially have this meeting and two others before we need to be ready for IAW and Connect-a-thon. Roots ID has blocked off, well, we've reserved a demo space for the demo time which is right in the middle of the conference. The conference is technically Tuesday, Wednesday, Thursday. Some proceedings before that that occur and maybe some meetups after that but that's the official conference time and so right in the middle of Wednesday essentially during lunchtime is the demo time and Roots ID has a slot for that and I guess we still have not said for sure if we're going to make that slot the Connect-a-thon time or if Roots ID will just focus on some of our stuff. Any thoughts on that? Preferences in terms of should we do it as part of a demo hour or should we? I'm talking about the Didcom Connect-a-thon as part of the demo hour or would you like to do an official IAW session as a Connect-a-thon? Do we know who's going to participate or we need to participate? Yes, great question. Great question. Okay, so let's bring up our page here and oh, yes, so Roots ID obviously, Ataloprism, Tony Rose has said that he will be there and certainly Roots ID can help with the Ataloprism side of things. Aviary Tech Brian had said that he wants to participate. Thomas, he demonstrated Nessus his progress there. That may be something that because I don't think that he said that he's going to be at IAW, so that may be something that we do like a recording of and show a recording of some connections between his work and Roots ID or anybody else who wants to connect with him. Ariel's going to be there and I think he's planning to show some things as well. I don't know if he said that he's going to do the Connect-a-thon? Yeah, no, he has another table reserved for that time, so as far as I see on the agenda. And I don't think he's ready for Didcom B2. Okay, all right, so I will let me modify that then. And Blocktrust Bjorn is going to be there, right Bjorn? Yeah, I would be there. So I would prefer joining your session or like your demo hour. Great, okay, that sounds good. I think do you know if Tony wants to show a demo or if he is just there? Yeah, he wants to participate. Certainly, we are working with them right now. Roots ID is working with them in terms of, well, yeah, continuing. Yeah, I think maybe we need to confirm that or ask him to reconfirm because I'm not sure he has the tickets and all the stuff he may forgot. Yeah, right. Yeah, fair enough. Yeah, I'm going to reconfirm everything because maybe people may change minds or whatever. Okay, all right, so we'll go through a reconformation phase. We'll say here confirmed and Bjorn, you are confirmed as well.  Anybody in touch with Brian to check with him? He had, right, just left the message, so I need to confirm with him. I'll consider that one of my tasks after this meeting to, yeah, reach out on Discord. Okay, so we have, let's see, two definite confirms. I suppose that Nessus, I mean, he won't be live there with us, but I think he's confirmed in the sense that yeah, he has the tools and he wants to at least show a recording of where that, oh, I messed something up. Look at this. I don't know, it's spread out all of a sudden. Well, I don't know what I did. Maybe these dashes are doing something. Well, okay, that's all right. Okay, going down the list, Pico Labs, Bruce. So we will have our bi-weekly meeting tomorrow. Okay. And I will ask the students, I've shared this ACMD with them, but haven't heard back from them. I know that neither myself nor any of the students are able to attend in person. Okay. But Phil will be there. I know he might not be there, but not representing this particular thing. Yeah, fair enough. So it would be helpful if I could identify someone who could spend some small amount of time working with the students to attempt to practice connecting. Okay. I don't know who might be willing to do that. Do you guys have, so you said your meeting is tomorrow? Yes. Is that something that someone could attend or that be too soon to connect with them that way? That's a good question. Or maybe at that meeting, could you discuss if there would be a certain time that's good to meet, and we could just see if it works out that, because it's hard to obviously coordinate a bunch of students and things like that. Yes, it is. I'm just looking at the calendar here. If it's bi-weekly, I suppose that it should fit before, you know, not your coming meeting, but the meeting after should fit before. Yes. And it's even possible if someone was free at 10 a.m. mountain time tomorrow, they could, we could arrange a meet. Okay. Yeah. Okay. I don't know on the roots ID side, I'll have to reach out and find out, but let's keep that in our pocket and, yes, certainly discuss with them tomorrow. And then, yeah, we'll see if we can get a meet up. Certainly the roots ID mediator is out there, the block trust mediator, I think as well is out there. And so that could be something they could just try to connect with and, yeah, I don't know how easy or hard that will be for them, but and they could begin to ask questions. So at the very least, asynchronously, I guess I'm saying there's a service out there waiting for them. So they could just start asking asynchronous questions as well. Okay. Thanks, Lance. I will bring that up to them tomorrow. Great. Great. And the link for the block trust mediator is also on this document. I have the roots ID one, two, let me move that up there as well, just so it's more front and center. And we have, yeah, a Jupyter notebook that I assume Roto did as well. Yeah. So I guess this section here, the provided services, you can see, you can see what's there. So have the students look at that stuff and then they can just start asking questions. Okay. Great. And then maybe we'll do also like a live meeting if we need to. Okay. Yeah, I think we should also add the mediator from Brahmo, that Nick mentioned. Do you have a link to that? I want to try to find it. Okay. And speaking of Nick, I do believe he's confirmed. He said he wasn't positive they'd be at IOW, but we know that Alex and he are in contact and working with each other. So I'm going to set that as confirmed because, you know, at the very least it's a similar story to Prism at the very least, Roots ID is capable of demonstrating Varamo and Prism as part of our demonstration. So yeah, I think that's sufficient at least. So, you know, at the very least it looks like we have, yeah, four, but probably five, probably six, and then Pico Labs, probably seven participants. And I don't know. I think I saw Alex. You can also consider as confirmed. It is confirmed that Mike and Char, Mike Ebert and Char Howland will both aviate IOW in terms of the projection. I'm not like, like I just attend these meetings to get updates for the general team, but they're sort of the ones in charge of that. But it seems like a green light on that. Okay. Can you confirm with them what they plan to do during the demo hour for IW? Or I can even reach out to them if you want. I can, Mike's a bit busy this week, but let me schedule a little quick meeting with them and I will get back to you. Yeah, or just, yeah, it's possible that, yeah, just, I mean, I don't know if they, I haven't looked at the demo list. It sounds like Roto has the demo list somewhere handy for himself. If they're not listed on the demo list, then I would think that, you know, then that would work quite well for them to kind of join us. And that would be cool. So I'm going to put them as, I'll put it here and also let me get there. I also wrote it in the chat if you need other names. And I assume it'll come up when you start typing. Also, I wanted to say cool background. Oh, thank you. It's my first time using it. So I appreciate it. Okay. So, yeah, good. And you put there right, Mike and Char. Is it pronounced Char? I don't know. I don't know. Char, not met them. She actually leads, I think there's a part one and part two of the Aries Working Group. So I think she leads every other week. Okay, cool. We're going to say tentatively confirmed. How about that? Sure. And I'll get in touch with the base up about that. Yeah, great. Cool. Thank you. Oh, I did it again. I think these spaces are what does it, yeah. Okay. Great. Okay. And then Animo, I know will be there, but I think that they said they don't know if they're going to be demonstrating. I think they actually, during that demo hour, they have something they are demonstrating. I can't remember what it is. Yeah, they have something, but also they, I think they don't have, they can't be too ready. Yeah. Okay. Okay. Okay, good. All right. So I think that's a pretty healthy list. I don't think that we are for sure saying that we're doing it during the demo time, but I think we're leaning towards the demo time being the connect-a-thon. That's my sense so far. Okay. Bjorn says, Alex, could you check the URL of the link? Doesn't seem to work. I'm not sure which link we're talking about. I'm maybe, oh, here we go. Yeah, but I'm a link. Yeah. Yes, got you. I see. That didn't come up as a URL in the chat, but okay. So let me copy that. And I will put that under provided services. Yeah. Good. Yeah, but it's not working. It's not working. Yeah. Well, I'm sure they'll get it worked out. Abramo, mediator is here. We'll be live soon. Okay. Good. Yeah, very good. I don't know how everybody feels about this. I don't, can't tell if this feels like much to everybody, but I'm very excited about this, you know, relative to what we did last IW. This is a really nice step forward. So I'm just really excited about it. Okay. Maybe we should go then through, in terms of required, we were basically saying, you know, hey, if you bring a service application or whatever that's did come v2, at the very least, you need to support out of band, basic messaging and routing. Is that still, we're still solid on that? Anything else we need to kind of say here? Okay, I'll take silence as applause. And then optional, we basically said, yeah, prerecorded videos. So that might be what we do with Ness's, for instance, I'd actually like to collect as many prerecorded videos as possible, because we could even do like, you know, just have it as something that's kind of running in the background on a laptop, but it's also something we could post to social media and things like that. So as much like recorded content, of course, we would also want to record during the session and add that. But yeah, basically, some kind of highlight video for all this effort. I really, I'm happy to, you know, kind of put that together as we get it.  And it can just be, you know, fairly raw and I can edit it down, whatever we need to do. Yeah. So anything else that you want me to list under the optional section? Okay. Um, I don't know, Alex. Was this still, was the OIDC for VC? Is that something that we still think is possible to show or should I just remove it? Okay. He says remove it for now. So I'll just take it off of there. We knew that was a stretch goal. I think maybe we save it for next time. Maybe I'll start a new section actually. Considerations for, it'll be probably November IW, November 2023. Yeah. No worries, Alex. He says he doesn't have his mic working right. I'm going to put this towards the bottom. Okay. In terms of our story and use case, let's be honest that just showing any form of, you know, interaction is like fantastic. So that's already like the dessert. The cherry on top is can we start working towards an actual, you know, use case that kind of everybody's aware of the use case. They're aware of, you know, possible roles that they can play. I see Alex maybe is unmuted, so Paul's for a second. Can you guys hear me? Yeah. Yeah, you sound good. So, sorry, I missed the, it seems, I don't know why there's two of me. I closed the other tab. Yeah, I see two of you, but no worries. What were you saying? Sorry. I do, yeah, I do see two of you, but no worries. And it seems to be working. Why did you ask? Sorry. I didn't catch that. Oh, I was just, well, I'm not sure which part you missed, but I was just saying that, yeah, we see two of you, but it's no problem. Okay. Yeah, it's internet. Yeah. So in terms of a story or use case, you know, maybe this is something that we save for, again, you know, the future. But essentially, there's this air travel use case that the W3C, you know, uses as a nice way to highlight some of the complexities of, for instance, VC, you know, well, identifiers and VCs. And, you know, well, here's a fairly complex use case that, of course, SSI wants to solve. So I don't, you know, this is just bonus material, essentially, when Thomas did his demonstration last week, this is the use case that he's following along with. And I think it's a nice one. So, yeah, just be aware of it. And, yeah, maybe gives, you know, all of us a bit of a frame to kind of work towards. But whatever various support you can do with that, that's all just bonus material, basically. Yeah. So group chat as well. I guess, I don't know, what is it going to look like when we're doing basic messaging here? For instance, yeah, of the, let's see, with the prism, you know, connect-a-thon activity that we did, it literally was kind of a simple chat view. I suppose that with someone like Thomas, with his command line stuff, I'm not sure what exactly it's going to look like when we are trading messages and how we're going to show that. Any thoughts on that? Do you think it's mostly like, for instance, Roots wallet, it looks like a chat, right? It's a kind of this chat interface. So do you think that with like Thomas' stuff, when we're interacting with his stuff, it'll probably be a recording. So we'll just be showing him doing the operations and then, you know, the messages appearing in Roots wallet. Do you think there's any other things that we'll be showing? Yeah, well, the basic is, I think, like that, right? Yeah. Just send the basic message and respond to the other receiver and reply back. Yeah, so like, for now, we're just trying to, for this interoperaton, are we just using the basic message protocol as like the client-facing protocol? Because like the aggregation, like just happening almost like in the background. Yeah, we're saying that required is essentially these three things. And so, yeah, the way to connect essentially, or yeah, the way to address one another and then the ability to basic message. So yeah, okay. All right, yeah, fair enough. And yeah, so there can be some level of explanation and then obviously some level of showing whatever we have to show in terms of our interface. I don't know. I haven't seen the block trust stuff. Bjorn, is there anything you want to say about kind of what you want to show on your side? Oh, I guess. So, yeah, I mean, it's a web browser. So you, I have a notebook. I put it there, open the wallet and you have yours on your phone and we can send messages. So that's the demo. Yeah, great. Yeah, and everybody actually could use it. I mean, it's there. That doesn't have to be scripted so much or so. Okay, awesome. Perfect. Okay. I think with the Veramo that Nick mentioned the other day that he will be able to, I think, understand the peer, but the message from the mediator is going to be in, Alice correct me, in this web, right? Or any, I think any of the Veramo supported the method. I do have a deep peer. I did create a deep peer plugin that works. Yeah, working. Nice. I'm able to create right here. Is that mean that all Veramo agents from here on out will have deep peer support? They can if they want to. Oh, like it's right. It's a plugin that they just need to configure. Yeah, fair enough. That's the perfect answer, of course. Yeah, good. Where is that plugin published? It's in our repo. Okay. Is that something that we're going to try to push out to be like NPM-able? Yeah. Yeah. I haven't done all of that work yet. I just make it work for locally. I need to do some refactoring for that to be NPM installable. Yeah, great. Awesome. But yeah, it works. I'm able to create the peers and with service endpoints, and I'm able to. Okay. All right. So that dovetails well with what we're talking here in terms of minimum interrupt profile. Is there anything we want to add here? I'm going to modify this to say test your agent service against any of the mediators listed. What a dynamic in the provided services. Okay. So let's just imagine these sessions are supposed to be in the demo, basically supposedly that people are walking around from table to table to table. And so this is a little bit different since we're doing a connect-a-thon. Well, I guess that's a question. Are we able in five minutes? I can't remember if it's five minutes or 10 minutes. I should know this. Yeah, whatever. Yeah. I think it's just a recommendation. Well, yeah. At least let me help everybody to know what the recommendation is. Okay. So the demo hour is, they call it a speed demo hour. And it's from 1.30 to 2.30. And you'll be going around to tables or participants will be going around to tables. So we'll essentially have a table that we're collected around. We're table number 11. Maybe I should put that in there. I should probably put all these details in here. Yeah. I guess the nice thing is that we don't have really to reset our demo. So that could be more of a constant demo because some parties might throw out their mediator and then reconnect with the mediator or some agent. So that doesn't have to be like the 10 minutes scripted demo. And then it's over. So it's a bit easier for us because it's more or less constant or can be at least. Yeah. Thoughts on that? Yeah. I see that the tables, the roundtables, so we may be seated with two or three laptops and a bunch of cell phones and maybe in the pairs and show anyone who appears here. We show what we're doing, how it works. Yeah. Different, yeah. Different actions. Good. Yeah. I mean, that seems good to me. I mean, it's pretty laid back and free flowing. Yeah. I'm just trying to think of if there's any more structure we can put to it that will allow it to be as successful as possible. I think somebody has to hand stop. Oh, yeah. Yeah. Go ahead. Yeah. Jump in anytime, Alexandra. Sorry. Is there any information where you got like the table numbers in the times? Like do you have a link for that? They sent me a custom. Well, I'm not sure if it's custom or not. Okay. Maybe never mind I know in our team who would get that. Okay. What did they send me? They sent me a Google Doc for my thing. The nice thing is they sent me a Google Doc that I can adjust because the way that I wrote it, I wasn't sure if it was going to be roots ID only and then we do a connect-a-thon in another session. So if we decide to do the connect-a-thon as part of the demo time, I can go edit the demo description to include all of this information, all the participants and stuff like that. I guess the link to our HackMD. So that'll be really good. I appreciate them doing that. Just looking through. They did give a link for the growing list of demos, but I'm not sure if you think Roto, that's something I can just, it looks like it's on the IW website. So let me post that here, just so that everyone's aware. Yeah, I don't remember how I got it. Yeah, it's on the website. Sorry, just trying to find my chat. It disappeared. Here we go. Zoom. I swear they make it as hard as possible on the hosts. Okay. So that's the kind of list of demos that'll be there. And like I said, I'll update kind of the description for ours. I don't think we're 100% sure we're doing it during the demo time, but we're heading that direction. Demo tables list. Yeah. Oh, actually I should be making that a link. Cool. Okay. Any other thoughts of how we can make this as successful as possible? I think especially things we need. I have a question. You said a couple of times, and I've forgotten the verb you used, but to try yourself against a mediator, can you unpack for me what that means? How do you try to use a mediator? Yeah, sure. So let's just look at the mediation page. I think it'll so this is just a page to give instructions for connecting. And so most wallets would scan a QR code, but there's also this out of band invitation that's here. And that is essentially how you would begin to do the connection with the mediator. So if you have a cloud agent speaking in conv2, you would make a connection to the mediator, and what would be the purpose of that? If you have a cloud agent, you will need a mediator. I think mediators are only for phones and edge devices. Oh, I see. So this is a way for you to register your phone-based wallet to use the mediator for incoming connections from other agents? Yeah, so that is the purpose. However, I did think that our mediator also can act as, correct me, Roto and Alex, but I thought our mediator had the ability to essentially act like a place that people can do a trust ping. I think we've even in the past used it to issue a credential. Is that, I don't know, thoughts on that? No, that's fine. Just for testing, you can use this out of band and send a trust ping, send a discover feature as well. I think the block trust mediator does do have these two protocols as well. And we also include like a basic message, a protocol that if you send a question, they're going to be replied by using responses from the Wolfram Alpha API that we found there. But at least a trust ping that is a really common protocol, the first that all the people implement. So you send a ping and you're going to get a reply back. Okay, thank you. So my understanding then is that a mediator is a cloud agent that speaks in COMB2 plus it has mediation capabilities beyond just being a cloud agent. Okay, thank you. Right, that's correct. Yeah. Can I essentially see that via the stupider notebook? If you use the notebook, you're going to have more complex protocols, but all those have request mediation. But you can start just with the trust ping and see if you got the replay back. And then you can go then with those mediation protocol. Yeah. Okay, so is there, do you think it's worth putting additional instructions here just to say something along the lines of if you just want to try a trust ping? Or do you feel like? Yeah, but yeah, I think there's something on, but don't get in the repo, but it would be nice to put there as well. Okay, got you. So somewhere here, maybe in a readme. Trust ping, the mediator also implements the following features that can be used as a playground to test other protocols. Trust ping, but this is probably just the definition of that protocol. Yeah, so that's only the definition that's supporting. Is there a link to the page that you're scrolling through, sorry, Rodo, is the link to the page that you're scrolling to in the markdown that we were looking at or? I'll put it there. Yeah, that's good because you can kind of get to it from the stupider notebook link, but I'll let's put that in here. Oh, thank you. Sure. Okay, good. What else? Other questions, thoughts? Yeah, I mean, it's pretty low key. I mean, whatever we make it is what it'll be, and yeah, I mean, it's good, but I would say maybe we should keep a log of how far we've gone with each thumb specifying that I guess it would be cool to know who's going to be able, who is already, which pairs of connections or basic, which pairs of basic messaging do we believe we've already demonstrated or are about to demonstrate, if that makes sense. Where do I want to put this? Document, recordings, yeah, probably here. Video interactions. Okay, and certainly when you're doing your sessions, if we can kind of capture where we're seeing failures, that would be great so that we can kind of update whichever side needs updating or both sides maybe of the interaction. Let's just say status log of rep for live demo. So we've definitely done roots ID to root ID and prism. That's happened. And really I should be saying roots wallet. We've done a quick question to that. So that's using the prism iOS SDK or how does it work? Because a cloud agent doesn't have endpoint for basic messaging. Right. Yes, this is with the Swift SDK and a sample app that they've put on top of it in order to give it a UI. And they use some kind of prism mediator or did they use yours? Yeah, they used ours. Okay. Yeah, good question. I also noticed that the prism demo had the connections protocol. Do we need to support that as well for other agents? No, I don't think so. Just if you need to work with the prism right now. Yeah, don't put that constraint on us. It's the problem. Yeah, I will most likely implement that, but I haven't started on that. But yeah, that's a goal to also work towards the demo of block trust and prism. Okay, so Alex, what is your intuition on kind of Ramo? Yeah, like we can send basic messages to them. I can try to implement the connections protocol from prism in there. But yeah, I'll let you guys know how that goes. I haven't implemented a outcome protocol in the Ramo agent. Yeah, I know that they have basic support for mediation protocol and basic messages. Okay. Like out of the box. Okay. So that's going to be the root wallet Ramo? What do you say, Alex? Yes. Okay. Okay. I mean the root wallet version that the Ramo root wallet, I'm gonna say. Yeah, so that's actually a good question. So there's kind of the idea of root wallet using Veramo's framework, but then there's the idea of obviously connecting with a Veramo mediator or whatever. I mean, obviously we'd have to, well, yeah, I mean the Veramo mediator, but then also just a, is there, if Nick is present, do we think that he's going to have some identity wallet or something similar or some form of interface? Yeah, interface or even a cloud agent or something that he's going to be working with. Like in terms of, I know that he owns the Veramo, like he manages the Veramo mediator in terms of services, but I don't think that they do have a wallet or anything like that. Okay. Yeah, like root wallet is the interface for... Yeah, okay, fair enough. Okay, good. It sounds like Bruce is looking to do Pico labs to mediators. We don't have to say necessarily which one it'll be, but good. Let me come back. And you are mentioned the blog to us, please. Yes, absolutely good. Thank you. Yeah, thanks. It doesn't work yet. You can put it under when and I'll record next week. Yes, certainly. Well, I should probably instead of be doing this as a bolted list, right? It should be some kind of table, but my table skills, I don't want to try to do this in front of everybody, but yeah. Anyways, maybe I'll convert this to the table so we can just give checks on who's going to interact. I mean, just put it below under when. Oh, because it's not there yet. I see. Yeah, thanks. Mer. Okay, good. And then let's go back up to our list and write Nessus when Nessus. And I don't know, for instance, I'm going to list Nessus roots wallet here, for instance, but I don't know, Bjorn, if you're planning to try to reach out. I was hoping that these sessions of today being kind of, we're doing a lot of coordination, but I'm hoping that next week, for instance, if Thomas is here, we're literally trying these things, right? I would love to, for Pico Labs or Varamo or anybody that we're just live together in the next two areas, did comfy two sessions, just trying to make connections and well, trying to do trust pings and yeah, everything, anything that we think is possible or that we'd like to be able to show. I would love us to try it live and see how hard it is. Dress rehearsals, I guess, is what I'm saying. What does everybody think about that? Do you feel like that's the best use of that time for the next two weeks? Okay, I'm going to take that as yes, silence always means you're all very impressed by my ideas. Completely agree. Okay, so same with, yeah, in DCO folks, if they ever have time, Alexandra to come visit during this time or, and we can always set up obviously sessions outside as well that might fit schedules better. I'll reach out to Tony for sure to see if he's going to come. Okay, yeah, Brian, I'll have to reach out to. Okay, so I think that's, I'm going to put, I'm going to list Brian down here. Alex, I don't know how, how, if you've talked to him recently, if you have anything in terms of where you think he is? I have not. I haven't talked to Brian in a minute. Yeah, he's been super busy with something, but I can't remember what, something, something out in the SSI open standards open source world. Okay, so I'm just going to put that like this. Yeah, okay, good. So just kind of this is going to be two, we'll call them apps, it'll be two did come, the two apps, the mobile apps, I should say, right? Or I guess we could be showing, well, we'll see, we'll see what Tony says. Yeah, I mean, for sure, we'll be showing kind of two did come v2 mobile apps. This is a web app with a mobile app. Correct. Yeah. And then this would be a web app with depends on which, which prism we're talking here. Yeah, I guess the iOS version then. So basically another version of your wallet. Okay. Okay. Oh, so you're talking, well, yeah, would we want to show block trust to the prism app? So or does the agent answer for basic messages, the prism cloud agent? Yeah, they have a demo with a basic so it should Okay, check. And we can call the guys also to adapt. Yeah. The only thing that, yeah, when you start working on that, we should check that issue that I found with the I think the last library on the Cibala last last library. Yeah. So we need to check that because the first prism use the last library. So that's okay. Also lands for the veramo they do. I'm just running their, their agent locally and they do have like a chat interface, like for a web app. Oh, nice. They're like web interface with our wallet. Okay. You said, I'm sorry, was it web or mobile? Yeah, it's a web app. Thanks. Okay, good. Yeah, very good. Okay. Pico labs. These are IOT type. It would be considered a cloud agent. Okay. Yeah. Okay. And these are would be acting as cloud agents as well. Okay, good. Nessus also would be like a cloud type agent. Okay. Yep. I don't know if it's obvious. Yeah. Okay. And then Brian, I mean, you don't have to, we don't have to know. I can reach out to him, but I don't know. Does he have a mobile app or is it a cloud agent? We don't have to know. I'll ask him. Okay, good. Yeah. I think he's running from the computer laptop. It's like a Node.js command line or something like that. Okay. Good. Okay. And this was an app as well. Okay. Sorry. I've forgotten what we said we wanted to do, which prism did we want to do for BlockTrack? I mean, in general, we could use as you said, those with SDK with your UI. But as Rodolfo mentioned, there's some issue with the Rust.com library. Okay. We have to figure out. Okay. Gotcha. Yeah. But maybe when I can talk to the cloud agent, that would also be fine, I guess. Okay. Okay. That's pretty much our time. Are these sessions helpful? I know this isn't standard. Usually we're discussing things happening out in the community, but everybody good? Yeah. Okay. I see a thumbs up from Bruce, Bjorn. Great. All right. Everybody's cool with it. So essentially, we're hoping next week that we start to demonstrate some of this stuff together live, or at least continue to make progress towards being able to do live interactions. And hopefully, that's yeah, a nice warm up. And we can record some of it so that we have some video in case people can't make it live. Okay. Great. Thanks, everybody. Fantastic. And we'll we'll see you next week. Thanks, everyone. Thank you. Thank you. Bye.


Ok, notre prochain Speaker est Philippe Poirotier, de Vagin, et il travaille pour vous, le contrôle, qui est en charge de 2,5 millions de lignes de code de Vida. 2,3 millions de lignes. 2,3 millions de lignes, chérie. Oui. Bienvenue à la 2e place, vous avez recusé un peu. Ok, donc aujourd'hui je vais parler de la performance de la track de la application de développement pour les opérations. En fait, ce que je vais parler n'est pas trop relative à Vida. En fait, il y avait même 0 slide dans ma présentation, contenant une ligne de code de Vida. Donc j'étais un petit peu affrayé que l'organiser de la salle de dévouement me tuerait après la présentation, donc j'ai ajouté un slide. En fait, j'étais connu quand j'ai dit qu'ils me tueraient après la présentation parce que Jean-Pierre, l'organiser de la salle de dévouement, en fait, il travaille pour moi parce que je lui paye pour soutenir le contrôle de la salle de dévouement que nous avons utilisé pour vérifier les règles de code de la salle de dévouement. Et l'organiser de la salle de dévouement, en fait, aujourd'hui, la 2e salle de dévouement est en train de travailler dans mon équipe, donc il y a des raisons de ne pas me tuer mais peut-être, bien sûr, le dévouement peut avoir des raisons de me tuer parce qu'il est en train de travailler pour moi. Voilà. Let's switch to the next slide. What are the objective of performance tracking? So the objective of performance tracking is to evaluate major resources needed by new functionalities that's one objective. Another objective is to verify the estimated resource budget, CPU, memory of what you develop. Sorry. So we want also to ensure that the new release will cope with a current or expected new load and we want to avoid performance degradation during development. For example, imagine that we have a team of 20 developers working six months on a new release. That's about the size of the team working on this application. And let's imagine that each developer integrates x changes per month and if one change on x degrés a performance by one person then optimistically after six months we have a new release which is 2.2 times slower. We start from performance 100% six months 20% one person. That's an optimistic view. The pessimistic view is that the new release is 3.3 times slower. We start from 100% times 1.01 to the power of 120. So clearly we have to do something and we cannot wait the end of the release to check the performance and see where the degradation is coming from. And so the objective is to do a daily track of the performance during the development. So we have a development performance tracking objective which is relatively precise because we want to reliably detect performance difference of one person or less. So a little bit of explanation about Euro control. So a little bit of explanation about where I'm working and what we are doing. So Euro control is the European organization for the safety of air navigation. So it's an international organization with 41 member states. We'd like to draw your attention on the fact that it is 41 and not 40 since last midnight. We have several sites directorates et cetera. The activities we have operations handling management of the traffic in Europe. We do concept development. We do some European wide project implementation and support et cetera.  Where I'm working is the directorate of network management. It has nothing to do with network routers, cable and so on. It manages the network which is used by the flights to fly. So we have routes, we have points, we have air routes. This is a network and that's what we manage. We develop and operate the air traffic management network. So we have different operational phase. We have strategical. We prepare the network, the restructuration of the network and so on. Months or years in advance. Pre-tactical, a few days in advance. Tactical, today, tomorrow, post-operation. We have to analyze what has happened and maybe improve for the future.  So if you have an airspace control center and you have too many flights that are planned to fly through this airspace control center, the controller might be overloaded, which is not good for your safety. And then the idea is that all systems will detect this load and then provide mechanism to balance the capacity and the load. So network management, we have two mission safety critical systems. So these are IFPS flight plan processing and ETF mess flow and capacity management and my team is developing these two applications which are quite a lot of code in EDA. EDA is by far the main language used. We have something like the core of the system is 99% EDA. So some more words about IFPS and ETFMS. So it's big application. The core is 2.3 million lines of code. And then around, we have shell scripts for the build infrastructure for the monitoring. We have a lot of test code around, etc. So in terms of capacity, let's speak about a big day of ETFMS, the flow management system. So in Europe, we have about in peak, more than 37,000 flights to Wendel. And we have around 8.6 million radar position. And these are plan to increase to 18 million in one year because we will add new sources of position, namely the ADSB data provided by new systems. So we have from external users, so the aircraft operator, France, British Airways, et so on, the airports, the air traffic control center, they all query our systems in order to have data. And this results in about 3.3 million queries per day in peak. And we are publishing changes on the flight data through public subscribe mechanisms such as AMQP. And we have more than 3.5 million messages per day to publish. So what hardware do we run on? So the online processing is done on a Linux server 28 core. So it is not a small server, but it is by far not a huge setup. We have some workstations that are running a graphical user interface for our internal flow controllers. And on these workstations, because they are quite powerful, we are also doing some batch processing and some background jobs. So we have many heavy queries. We have complex algorithms that are called a lot. For example, we have queries such as cons or give me a flight list for all the flights that are traversing France between 10 o'clock and 20 o'clock. We have some algorithms like lateral route prediction or route proposal optimization. We have vertical trajectory calculation. I'll show after some drawing to show a little bit the complexity. And a lot of other things you can imagine in 2.3 million lines of code. We do a lot. So this is a graph of a flight departing somewhere in Turkey, arriving somewhere outside of the European Union. And you see that when we have to calculate a trajectory. So for example, these orange stuff that you see here are radar position. And each time the system receives a radar position, it has to detect if it is far or not from the planned trajectory. And if it deviates a lot, it has to recompute the trajectory. So this is just computing one or several routes if the aircraft operator changes the flight plan for a flight. But what we also provide as a service is a route optimization service. So they can call our system and say please find me a good route. And then we have to search maybe this way, maybe this way, maybe this way. Of course, basically, it's a shortest path algorithm d'extra. But we can't use purely a d'extra. It's a lot more complexe than that because there are constraints which cannot be modeled in a simple graph. And so even searching, let's say, the end shortest route is a lot more complex than typically than a typical similar problem. This is the same flight that we have seen, but on a vertical view. We see departs here and we land there. And we see that we have plenty of constraints like forbidden airspace. We have some levels that have to be respected. We have blocks of airspace that can be traversed or not, depending on the direction and other condition. So you see that our algorithms that we have to make are quite complex. For this algorithm, for example, the algorithm that search where the flight can fly, several hundred of millions of calculation to do per day, which are done either on the central server or in batch or background jobs on workstations. So performance needs and it's scalability. So as we have indicated, we have a lot of users. We have a lot of queries and so we need horizontal scalability for this or operational configuration of our main server because we have a lot of instance of this software because we have, for example, an instance of the same system which is doing the prediction for the next week. For example, we have standalone systems that are used to study what happens in several months. Here I'm speaking about the operational system. So on our 28 core system, we have ten high priority server process that are handling the critical input. So the flight plan, the radar position, the external user's queries. We have nine lower priority server process having each four threads which are handling lower priority queries such as find me a better route for flight Air France 123. We have about up to 20 process running on workstation which are executing this batch job or background queries. For example, every or search a better route for all flights of aircraft operator British Airways departing in the next three hours. And that, of course, is quite a high load. We also need vertical scalability. So this is a little bit different of systems where you can have a lot of users asking a lot of queries and you can distribute it, and that's what we do. But we also have some functionalities like simulation where one single user needs a lot of power because we have our internal flow controllers. They have sometimes to take heavy actions like if an airport calls, or an airspace control center calls and say we have a technical problem and we have to take maybe 1,000 flights and we have to find how to best handle these 1,000 flights. All these actions will be done on the complete set of flights in the system. And so we must provide to our flow controller a functionality which goes fast for a lot of changes. So we need vertical scalability so something very fast. For example, to evaluate heavy actions such as we close an airspace, we close a country and we have to spread the route or delay the whole traffic impacted. So to give an example, starting a simulation it implies to clone the whole traffic from the server to the workstation.  And we need to recreate the in-memory indexes which are needed to execute all these algorithms and so on. So it's about 20 million in-memory indexes that we have to recreate. And in the release, we are busy developing. We have spent quite some time to optimize a start simulation. We are now starting a simulation in less than four seconds including bringing the data from the server. And this is using multi-treadings. So we have one task where flight data receives a stream from the server. One task creates a flight data structure once it is decoded. And we have six tasks which are recreating the indexes. Ok, so what we have seen now is that we have high performance requirement and we can't degrade. And so we have to track performance during development. And one thing we can use for that is performance unit tests. So performance unit tests these are useful to measure things such as basic data structure, hash table, binary trees and so on. For example here you see that we have performance unit tests which is checking the speed of an insert in a balanced binary tree. And then we can with this double check that for example the N log N behavior that you expect is effective respected by your implementation. So we can use the performance unit test to check the performance of low level primitives such as p-tread mutex aida protected objects etc. So this is performance unit test which is verifying various things. So it checks the low level p-tread calls that we are available on Linux. We can compare with a protected object on higher level aida concept for this kind of things. We have also some timing here in this unit performance unit system called clock get time clock monotonic 40 nanoseconds for this one clock get time but for the thread CPU time about 400 nanoseconds. Interesting to remember these figures because I will speak about it a little bit later and the difference between these two. So we can also use performance unit test to evaluate measure and be sure that we have the required performance for low level library for example malloc. In aida we don't use malloc directly we use the aida language allocation de allocation but at least with Gnat it is based on calls to the underlying malloc library. So performance unit test they have a lot of advantage they are usually small they are usually fast and they are usually reproducible and precise. Remember a one person objective. If there is a degradation we want to detect it and we want to detect with a one person objective. No, we have some pitfalls with performance unit test and I'll describe a real life example with malloc. So malloc performance unit test we have developed a performance unit test to compare the g lipset malloc with tc malloc. So, 7 years ago we switched from the g lipset malloc to tc malloc because we had less fragmentation and it was faster with tc malloc all fine, all good but when we parallelize the start simulation where we had for example to recreate this 20 million in memory indexes we saw some not understandable 25 person variation so sometimes start simul was taking 4 seconds and sometimes it was taking 5 seconds and this we saw that the performance difference was varying depending on linking a little bit more or less code in the executable but this code was not called and so the size minimal change to the size of the executable were causing difference and so we said let's analyze where this is coming from and we started to analyze this valgrin call green to really see in detail the instruction executed and we saw no difference we use the linux perf tool to analyze then what was the behavior not under the valgrin call green simulator but the real stuff and perf shows effectively that the tc malloc slow pass was called a lot more when we had maybe 10 bytes more or 100 bytes less of executable code not called we couldn't understand this mystery we saw it was so it was more often in the slow pass but we couldn't determine why so we said that's easy now we will re-measure the malloc library so we do a malloc performance unit test and tasks simulating or indexer tasks doing a million malloc and then a million free what have we seen we saw that jalyp say was slower but had consistent performance with this unit test je malloc with this performance unit test was significantly faster than tc malloc we were really happy but when we use je malloc with a real code the real start simul with a complete system was slower with je malloc so what was the conclusion more work needed on the unit test so that it better simulates what we have to do so we continue to work on the unit test after improving the unit test to better reflect the start simulation work what have we seen tc malloc was slower with many threads but became faster when the unit test was doing L loops of simulating start top simul so you start the unit test tc malloc once you start the unit test start simul once with je malloc tc malloc slower but when you say to tc malloc start stop, start stop, start stop then it was faster and not ok with je malloc we have observed that then doing the m million free in the main task was slower and that in fact when we stop simul is a main task that is doing the free the unit test it does not evaluate fragmentation and I have not put all the mysteries that we have seen with this unit test but still based on what we have measured we obtain a very clear conclusion with this unit test and what to do about malloc conclusion is that we cannot conclude from the malloc performance unit test so currently we have decided to keep tc malloc and we will reevaluate with a newer jlipset in red at 8 we are currently on red at 7 because on red at 7 the jlipset is quite quite old ok so pitfall of performance unit test as we have seen is difficult to have a performance unit test which is representative of the reload malloc we obtain in no conclusion the pit had mutex timing that we have seen it was a very simple measure in fact we have measured without contention but what should we do we should maybe also measure with contention and what type of contention and so on and if we have to to have a unit test representative of the reload what kind of contention do you have in the reload and if you measure and you see where you have contention and you simulate this in your performance unit test that might be for your current release but if you change your code the pattern of contention might change and so performance unit test are nice, small, fast but it's difficult to have them representative even for hash table, binary trees and so on the real behavior depends on the key types of the hash function, of the compare function of the distribution of the key values etc so if it is already difficult to have performance unit test for such low level algorithms what about performance unit test for more complex algorithm for example how to have a representative trajectory calculation performance unit test you remember the picture at the beginning how do you do a performance unit test for that, with which data how many airports, route air spaces with what flights a lot of short all, a lot of long all flying where you might have a lot of variation in the data so conclusion on performance unit test these are useful, somewhat useful but they are largely insufficient and so for this the solution is to complement in fact to do most of the performance tracking not with performance unit test but to measure and track performance with a full system and the real data so we want to replay one day of operational data so replay operational data the operational system ITFMS, it records all its external input so it records the messages that are modifying the state of the system the flight plan, the radar position etc it records the query messages you know the flying in front end systems don't really ITFMS recording it for example query such as flight list entering France between 10 and 12 which might be asked by France for example by the France control center and so we have a replay tool ITFMS has a replay tool which can replay the input data so of course it means that the new release that we are preparing we must be able to replay the somewhat recent old input format so this brings a little bit of constraint on the development and with this we have some difficulties we need several days of input to replay one day because you can have flights that are filed several days in advance you see the flight plan might have been filed two days in advance and so we have to replay more days no what is the elapsed time that we need to replay several days of operational data this is a problem of course what is the hardware needed to replay the full operational data well we have seen that we have let's say a medium size server and work station if a developer wants to evaluate the impact of a performance well we have to ask then quite a lot of hardware so that's a problem to replay the full operational data no also we want as you remember our objective of one person or better how to have a sufficiently deterministic replay in a multi-process system multi-traded system this is quite a challenge and we will describe it a little bit after remember one person so volume of data to replay so replaying the full operational input is too heavy and so the compromise is to replay the full data that changes the state of the system flight plan, radar data etc but to replay only a subset of the query load so we replay only one hour of the query load of the real system and even in this we replay a subset of the background and the bad jobs we also have a problem that replaying in real time is too slow as I've said if we have to replay several days to have the result of one day if we replay in real time it means several days if you want to do daily tracking of the performance you have a lot of replays that will be in parallel a little bit everywhere and so we have to try to reduce the time needed to replay but we can't just take all the input and replay it instantaneously as fast as possible without doing something because an input must be replayed at the time it was received or not if you have a flight plan at 9 o'clock and a radar data at 9.30 and another one at 9.35 you can't replay the radar data together with the arrival of the flight plan you have to wait till the radar data is a correct time to process it so many actions also happen on timer events and so what we need is an accelerated fast time replay mode so what is it, the replay tool controls the clock value and the clock value jumps over the time periods with no input and no even and so we are processing the data at the correct simulated clock but when there is nothing to do the clock jumps instead of doing nothing so the fast time mode with all these limitations on the data and with this time control we can no replaying one day and the data needed for one day in about 13 hours the fast Linux workstation we don't need anymore a big server and we can use a workstation which is not so huge still we have to take attention to the source of non deterministic results so one source of non deterministic result is a network, DNFS and so on if the database, files or whatever are on the network then the replay is not the only user of the network and that can introduce quite a lot of variations much bigger than the one person we want to detect so the solution for this is very easy we replay on isolated workstations they have their local file system their local database and so on now another source of non deterministic result is a system administrator you might say but how can a system administrator give non deterministic result on a replay, well we use to audit to see what's happening whatever and these jobs if they suddenly intervene during your replay it changes the performance and so the solution is to discuss with a system administrator so that we can disable their job on the replay workstation and the solution was not too difficult another source of non deterministic is the security officer they absolutely want to be sure that there is no virus no route executable not allowed and so they also want to run audit jobs and scan jobs and so on and here also the solution is to discuss with them but it was a little bit more difficult we also see that non deterministic results are obtained because of the input output past the story you start a replay from scratch but because you use files and a database that was already used previously even if you removed all the files if you clear the database you still do not obtain exactly the same and this was annoying us with a one person objective and so removing files clearing the database was not good enough and so before each replay we completely recreate the file system and the database for each replay even with that it was not ok because the operating system usage story itself if you do twice two things one after each other on linux the second time it might have been the memory in the kernel whatever might have been changed and so what we do is we recreate the file system the database and we reboot the workstation before each replay with this we still have some remaining source of non deterministic result for example the time control tool we serialize most of the input processing most but not all for example because if we would serialize everything it would slow down a lot the replay for example the radar position that have arrived at the same second we are not replaying them one by one we have several process and the several process will process the radar data that arrived at the same second in parallel and that can introduce some non determinism the replays are done on identical workstation same hardware as I've said file system database recreated, rebooted still despite same hardware, same operating system restart from scratch and so on we are observing difference between workstation so all workstation are not born equal that's our conclusion small difference in the clock of the CPU or whatever but we see we see some impact with all these caveats and the limitation and all what we did we have finally achieved a reasonably deterministic replay performance with 3 levels of results we have a global tracking where we track the elapsed user and system CPU for a replay for in the complete system we do a per process tracking user and system CPU and some per stats recording and we have a detailed tracking and one of replays under valgreen call green this we run on the side because this takes quite a lot of time it's very slow it takes 26 hours but it is very precise so this is a drawing these are all the baselines that we are building so we do continuous integration every day developers are integrating we are building and replaying and this is the release tracking of the global performance of the release we are developing where we have done this optimization for the start seaboo in green you see the total user CPU of all the process that are replaying in blue it is a total system CPU for all the process and in red it is the elapsed type so what can you see is that we have seen a very gradual improvement of the performance during the development and this gradual improvement but we have managed to track to see that the optimizations that we are doing were effectively optimizing or we were able to see performance degradation on a global level using this by the way this and this and that is not that suddenly we got a quantum computer that did the replay in zero point something second is because we had a problem during the replay of course I'll discuss more in details later on this part of the graph remember a little bit the pattern so that's a global tracking we also have a per process tracking where we record the user system CPU the eep status how much was used 3 tc malloc details because we are using tc malloc and so on so that's a kind of thing that we will record for each process so that we can see what's happening here we see four process which are processing fly there is the one which is processing flight list and cons and so on and for each of these process we are recording data that allows us to understand which process if we see a difference on the big global graph which process has increased and then the third level is when we have to analyze what has happened inside the a process then we have a one or of replay under val green call green and then we use the excellent kcash green tool and the excellent call green tool to record the call stack who has spent what we can see the functions which have consumed the top most and we can see the code which jumps we have which condition was called was often true or false and we can go up to the assembly language level so this is the main tool we are using when we want to optimize optimize some specific algorithms or when we see a degradation by the way I am also the organizer of the debugging tool deaf room and tomorrow there is a talk about at least one talk about val green so if you are interested that was the advertisement so another interesting thing to discuss is that what we measure is we want to avoid performance degradation but we also want to see that if we believe we are doing an optimization is that really an optimization and as a real life example of what we believe was a missed optimization something that we could optimize that we try to optimize and then it was becoming a failed optimization so this is the slide with a little bit of data code I promise to Jean-Pierre we would have some lines of data code here they are so here what we see is a data task and it has two rendez-vous so this task is maintaining the automatic loading of data like when the airports are changing it synchronizing the access and the load of this data so while someone is accessing this data it must be locked and when it is not locked the task will load new data in the memory and so this task among others has to maintain the number of locks and so it accepts so it has a rendez-vous which is called unlock where when a client task says I don't need a lock anymore it's calling unlock and the task decrements the number of locks and there is also a rendez-vous called getlock count which returns a current lock to the client what is it used for this is used for because when a process is activated to handle something like a flight plan it's not supposed to have a lock anymore and so at the top level of the processing we are checking that there is no lock by calling this this getlock count ok a rendez-vous a rendez-vous is a data task is something which is relatively costly because it is a task calling between quotes another task so there is a task which there is some system call to synchronize which is relatively expensive so the optimization idea was to decrease the number of rendez-vous by using lower level synchronization based on volatile that was the optimization idea the idea was to take locks not as a task maintain variable but as a global volatile variable we have a function that returns a value and inside the task we know accept unlock and then we are doing the decrement in the body why do we have to do it in the body while above the unlock the decrement was done outside of the body because imagine that you have the task which is processing my flight plan it has finished it releases the last lock and just after says no I will check that there is zero lock here when with this solution we must absolutely be sure that when the task that says unlock gets the lock count that it must have been done and so we must do it in the body because otherwise possibly we could have a risk condition in our check so this should be faster because we will have the same number of unlock rendez-vous but we will have much faster get lock counts accessing a volatile variable than a rendez-vous and so this is supposed to be much much faster that was the idea in reality we detected with the performance tracking that this was a pessimization this thing the compiler, the EDA compiler is quite efficient and helps us to build high level synchronization algorithms an efficient way and so for example if you have a nobody rendez-vous then the compiler will optimize this it will be a lot less costly to have a nobody rendez-vous than to have a body in the unlock calls and so because of this becoming more heavy in fact it was a pessimization and this was detected this is the extract of the big of the big drawing at the beginning so this was without the optimization here we had the little bit of problem and here was with the optimization the system time slightly increased so here we start to say what have we done during several release and after we have said we better roll back and so here we have rolled back the optimization so that we are we are back at the original system time performance so what can we conclude about that is that you need to track performance you need to track performance because otherwise you can have problems you have to track performance of your optimization because otherwise it might become pessimization and a third thing to remember is that you can believe you are maybe smarter than the compiler but it's difficult difficult but not impossible because here you see we did the optimization correctly not using volatile but using atomic and doing a part of the the decrement operation was done outside of the task and so it all went okay so we detect here that we can optimize something we optimize but it is a pessimization we ask ourselves why are we so stupid we understand it and then after we become more intelligent so that's quite positive as an evolution so performance tracking a summary we have good depth performance tracking using a mix of performance unit tests replay operational data as deterministic as possible the replay day we have to change it because the pattern of usage might change we might have new companies appearing airspace being restructored new route old route and so on so we change the reference day that we compare relatively frequently so that's too much new usage pattern we use various tools Valgrin, Colgrin, Cachegrin, Perth, Top and so on but we have to take care about blind spot of your tools for example Valgrin, Colgrin, Cachegrin it's very easy to use it's really a main tool we are using to optimize but it is very slow and it serializes multi-treaded application and so it measures something for contention for example you never have any contention when you run something under Valgrin, Colgrin and it had a limited system call measurement it was not measuring the number of system call and the elapsed time in system call but not the system cpu and we saw here system cpu had to measure as I also happen to be a Valgrin developer I change Valgrin so that now it also measure the system cpu spend in system calls so we need to have global indicators we have to zoom on the details where needed and as I have said improvement in the pipeline the next version of Valgrin, Colgrin will measure system cpu and we are also working on developing a Colgrin to help visualizing the difference because currently comparing the K-Cache green graphs is a little bit difficult so it looks wonderful we have a nice tracking system nice graph we can measure from the global system to the details of the process and can we be happy with that is that good enough to go operational what about you are on call or me I'm on call from time to time and you are waking up Saturday 4 o'clock because the users are complaining that the system is slow well I can't reply I will replay tomorrow Sunday the day and on Monday I'll explain you why we had big problems on Saturday this is not acceptable we have other questions like is the reference day that we replay is it representative of what happens on ops what about evolution of the ops work load and capacity planning for example if we say we believe that our users will do a lot more queries to optimize their routes because we have improved because we have new users that will say we want to use our service will the system cope or do we have to change the hardware setup do we have to upgrade the hardware put more hardware for this we need something else than the replay for example what additional capacity hardware capacities need to support each time more queries of that specific type the solution for this is to have permanently activate response time monitoring and statistics so this I'm speaking about the tactical response time tactical because it's mostly useful during tactical operation but we of course also use this during the replay the idea of this is that the application contains measurement code at critical point such as every removed procedure call invocation begin and end so one process is invoking something in another process it will measure how long it took at the process side that execute the removed procedure call we will also measure how much it took to process and send back the reply we are measuring the database access time begin end of the database and see if we can algorithm begin end such as calculate a vertical trajectory and so on so the measurements are typically nested for example inside an RPC execution begin end we will order begin end for sub things that are being used by the thing so this tactical response time package maintains a circular buffer with the last end measurement and for each begin end measurement it records the elapsed time the thread CPU time optionally the full process CPU time you remember at the beginning I said clock get time monotonic and clock get time thread CPU this is measured with clock get time monotonic and this is measured with clock get time thread CPU this is relatively heavy costly this is in fact a virtual system call implemented for those in the VDSO of Linux and this is really switching to kernel to get the data so if they are kernel developers in the room if you could improve the clock get time thread CPU that would be really nice so we this package also maintains statistics how many measurements of what kind were done an histogram of elapsed thread CPU and details about the end worst case this is giving a reasonable overhead about 1.7% of the CPU is spent in measuring what the application is doing rather than doing real work and for this we largely prefer to have it always activated on ops on our test and replay and so on because this is critical for us to understand what's happening in our system so this is an example of we have an online access to this data structure so this is a screen which allows to look interactively if you are called for example during the night what's happening and you see here a kind of tree a kind of tree of actions where we have received a flight plan we had a flight deviation then we did several derivation phase for the calculation of the trajectory we read the flight in the database we calculate another concept finally we have distributed some data and we have we have committed at the end the data in the database and so we can track the details of what's happening in the last end measurements this is the statistics that are maintained so we maintain how many measurement we had sorry the total time spent elapsed field thread CPU the average and a distribution in the worst case in case something is really abnormal it will appear in this data structure so this tax response time as I have indicated is used from development to operation on development it tells to understand how the system works see message exchange between process the algorithms executed the statistics are used to analyze the performance replay we can compare using this tax response time we can measure the resource consumption for new functionalities because we are recording things that are need to be recorded and so on ops online investigation of performance problem bug investigation in our system in our code the policies that exceptions are used for bugs not for normal behavior and so if we have an exception we take a cordon we can take a cordon without stopping the process we take a cordon we drop the input that has created the problem and we process the next message and the cordon contains the full status of what the thing was doing including in the tax response time the last M measurement of what the process did because possibly the bug might be created by something that was done a little bit before and so with this we can see what the process was doing and recently did we also use this for post ops analysis and thread analysis and input for our capacity planning so performance tracking of a big application a summary so we have a reasonably deterministic performance tracking during development it allows us to detect performance regression on a daily basis we can verify that the optimization that we believe our optimization have the desired effect it allows us to plan capacity upgrade for demand growth and new functionality etc we are using a mix of various techniques and tools such as performance unit test, replay, real data application self measurement we have to take care to avoid blind spots by using various tools perf, val, green, cold green, top s trace all will learn you something about your application the tooling we are also using for other purpose for example the replay tool is also a automatic testing tool because if we say we can replay 50,000 flight plans we can also inject 2 flight plans and verify that this one has this characteristic it has received this delay for example so the replay tool is also the test tool it is also used by your users to analyze, optimize operational actions and procedures if they did something and they say maybe we should have done something else they can replay of course on offline systems and then they can decide to try other actions what they did during operations so as I've indicated your operational system needs to have performance tracking and statistics this is not only for development voilà so that finishes my presentation yes yes typically yes ok so the question is when we implement new features do we inject some data in the replay in order to see how it behaves yes the idea is that during the development when we implement a new feature we are of course developing unit tests for this new feature which are done with the replay tool and whenever we have relevant data that we can use we might have to discuss with our users or we might have to create them ourselves we will measure how the code that we have developed behaves now it's very difficult to know what to do exactly because the real usage depends a lot of the pattern that the real users will do and sometimes users are a big source of non determinism yes most of the load yes ok so the question is I've said that it runs on one server and so how is this highly reliable yes I said when it is working it's enough to have one server but of course we have several servers we have cluster of server and if one server has an hardware problem for example we move the system to another hardware server so operationally it's good enough to have well let's say medium size linux server but we can move the application to another place yes in the yes no we are just taking a second measure so the question was how do we ensure precise measurement in the multi process so first the measurement that I showed where performance unit test so this we just have a single typically a single executable which is the performance unit test in the real system we just say this process has consumed this thread sorry this task has consumed this CPU to do this action that elapsed time no what we want to measure is what is really happening let's imagine for example that we have somewhere a kind of critical sections and two process are accessing some data and are looking this data you might have that one process has to wait a long time so that we will see because it says I see that I have spent let's say 0.1 second waiting and consume only one millisecond of CPU ah either there was not enough CPU on the test was overloaded or there is wait time and so we just measure what the kernel gives back via the clock get time system calls other questions no thank you all speakers