Hello friends Good morning, okay Let's see. I am a little bit behind here, so I haven't created our page yet, but we can do it in real time Perhaps Bruce are you there as well?  No no idea how a collaboration could Be possible maybe it would be nice to get An idea or sense. I mean, obviously pico labs is Supposed to be helpful for students who are, you know, learning about IOT and and and wanting to implement things, but I didn't know if you guys have kind of Partners and and like market use cases and stuff like that or if it's purely academic and I I mean no no disrespect by saying purely academic. Hopefully Hopefully that doesn't come off badly No, no, that's fine. It's It's mostly academic but It was used in an actual product for connected vehicles Okay, and was that like a Was that a Like a positive Interaction and do you feel like it was useful for pico labs as a as an organization or Yes, it was it predated pico labs per se Oh, okay So it was an it was an actual company that bill winley ran and He had I don't know dozens of clients Dozens of customers and it only ended because The manufacturer of the device that you plugged into the car Seized operations. Okay Yeah, fair roots ID knows all about Runways in terms of projects ending and You know work work discontinuing. So I understand that Yep, since then Phil has used it in a class that he teaches every year Gosh, yeah, so Okay, very good. I'm using it on an internal project As a proof of consent Okay There are a few dozen people who know how to how to set up a pico system Okay And Hirely fills former students Gotcha Telegram sand even knows how to do it. Oh, interesting. Okay And then can you compare it in terms of ssi Can you are you aware of other iot? projects Or related things and can you compare kind of pico labs to those other projects Yeah, so I'm a member of of an iot special interest group at diff And so but that group is working mainly at a That I don't really want to say higher level but at a level closer to Concepts and marketing and white papers and that sort of Yeah, to me my understanding is that a group is not that technical Heavy like the iot group at diff. Is that correct? Yes. I'm I'm probably the most technical person there. Yeah I have I have a little trouble actually bridging because I don't I don't think the same way They do yeah I've listened to to some of the meetings are some of the recordings and yeah, I was like more Business focused right in development focus. Yeah Trying to figure out where where ssi fits into the internet of things and very I mean everybody is is jumping on board Internet of things devices And we we would like to see them doing self sovereign identity of some sort Yeah Gotcha, and then any other uh projects that you know of um that are ssi iot related No, I don't really okay. Yeah, very good. Thank you Uh, okay, um, I think I saw yep roto hop into cool great. Uh, so today is the oh hi. How are you? Good welcome back He's he's still across the world Yeah, great Okay, this is the uh, February 13th 2023, uh areas did come v2 working group Uh, please be aware of the hyper ledger antitrust policy and the hyper ledger code of conduct And if you would like please add yourself to the attendees list Uh bruce you you were left over from from last week. So you're already on there Let's see and so was I by the way, um Okay, if we want we can uh, just go around real quick and if anybody has anything specific that they want to talk about in terms of Did come v2 related things? Um And then I'll I'm happy to go through the status of everything or if you want to wait till we go through the status. That's fine, too The only thing I have is that there is like the open wallet foundation has a did come v2 presentation today Yes, thank you. When that is uh, let's see That is The hour before the Diff did come use a group. Okay And uh, I'll be leading the diff did come uh, use a group. So any support and help uh, is much Appreciated so that that's a good convo, too But okay, that means today is full did come day for uh, for roots. I teach Yeah Yeah, open wallet Okay, good. Good. Oh Okay, you want some right? Yeah, yeah, I'll tag along and I haven't talked to Sam exactly how we're gonna break it down yet, but I'm sure We'll figure out a way. Yeah Yeah, basically, basically Sam wants alex to kind of be the hey, I've actually done, you know did come v2 and You know, I can It's not just the theory that sam, you know, we'll be presenting. So so alex is providing some, you know Tangible experience Yeah, so what I'm planning on doing is kind of like uh, they exactly speak a little bit a couple minutes It's about like my experience developing they come v2 protocols and then run through the cheaper notebook real quick and then Yeah, like do the basic message flow that we have between alice and bob and like show them like how relatively straightforward it is Once you already have public mediators available So we can kind of like also like mark it over mediator that way The jupiter notebook that you're referring to is from your jff activities or No, so the rodolfo has like a sample Like as a basic message between rodolfo between alice and bob and like a kind of like I Like tweaked it a little bit. Uh, so yeah, just like using that one just in the basic message We're like actually like issuing credentials and Anything like that way. Yeah there. Okay, great Um, okay. So, yep, uh, I'll try to attend that for sure. I'll at least be listening in And good Okay, anything else pressing We want to add to discussion topics Yeah, okay, let's go through status real quick Um aries agent test harness. Uh, I keep saying I'm going to work on this and I have done some but most of what I'm doing is um Trying to uh integrate an agent. Uh, that's a non aries specific agent In into it and well it took time and then I since have been Pulled off into other things So that keeps happening. Uh, but I guess that's how Test harness life goes sometimes. So I don't have uh really anything to report on there again The goal is to eventually start filling in aip 3.0 especially did conv2 related tests and to show Well for for me personally and for root's idea essentially to show this um Agent from from outside of the aries ecosystem being integrated. So That's all I have to say about that anything for aries ascar In terms of akapai, uh, you know, we haven't had how come back since uh, he's switched roles and Yeah, I haven't heard maybe I should ping him in the discord server and just see how he's doing uh, I I mean we really appreciated his contributions before but yeah, I also understand how Events ob e right overcome by events All right, it's I'll try to I'll try to reach out Okay aries framework javascript. I I still have not seen any movement on Um the pull request for did com v2 So that remains in on hold as well Yeah I have seen some movement in terms of oidc And it's possible that that's partly um being fueled by uh Ebsi, which is the european blockchain service infrastructure It looks like they uh are an oidc based communication ssi stack so, um, it's possible that afj is seeing movement in oidc for that but That's just speculation on my part Okay, uh picos As area agent yeah any updates for that bruce It was the off week for the students they meet with us every other week. Okay Tomorrow, so I have nothing new to report there. I'm hoping So just over did conv2 But uh, I'll be not tomorrow Okay, very good Uh, no, I haven't seen any movement uh for did come for the swift framework Uh, in fact the last thing I saw on that I think was a did conv one thing, but that was quite a while ago And then varamo anything on that alex you've seen for did conv2 Yeah, actually, so yeah varamo they last week, I think they merged mediation So now their agent will be able to do mediation and I was talking to nick And he's saying that they're gonna like publish like a mediator like a varamo mediator soon um, so Yeah, that's the update there Oh, yeah, I looked through their their PR and everything. Yeah, nice Okay, very good And yeah, I think um Yeah, roots id's mediator, you know continues to be useful uh that that uh, you know, I hadn't really, uh Understood how often that that might come up in different situations, but yeah having a mediator out there as you know seems to be uh, quite useful for the community, so This is good to see more mediators possibly coming online Is that a v2? Yeah Ah, we talking about varamo or roots id and that Yeah, both Well, maybe I can get the students talking to you Yeah, that would be awesome. Uh, okay, uh mobile. I don't have any updates on that. I haven't been attending those meetings lately. Just too busy Um, I haven't heard anything going on about the aries mediator service Any That's a good question Uh, I don't know Actually, maybe I assumed uh, it was v2, but that's probably a silly assumption giving given uh, what we know about the aries ecosystem Yes, I I expect this this call to be the leading edge of that. Yeah. Yeah fair okay how about for um Aviary tech there did come uh type script info Any any movement on that anyone seen? I was a minute. Oh, yeah, it's talking to brian. There's like, uh, he's been working on ydc lately Uh, so yeah, like no movement on the on the newcomer stuff Um, have we seen a good comparison between oidc and did come? I mean, uh Oh, we have like a riot w Yeah, have we seen like a published one that we can reference? Um Because uh more and more especially with the the ebsi Um, you know right now I'm being focused on oh i dc. I feel like I need to be well first on The differences. Oh i dc was new to me, you know until I came to i or until I was introduced to it at i w But I don't claim to have Well, uh, at least what what I know is i dc is server to client, right? So that's the main difference Yeah That's the server issue credentials and you are the client and that's it. But that's the only the only thing I know Yeah, so more web 2 Kind of model more 100% right Yeah, it's like a federal it's a Yeah, federal academic system It's like a federation type of model Right So so a bit uh apples or just comparison, but um because of But obviously, you know major overlap in terms of uh issue credential and things like that Yeah, it's like comparing a widec for vp and vc with uh issue credential and present proof Yeah, that would be like a better comparison. Yeah Okay fair enough Um, okay, uh carry Working group Okay, I think I can share the this the link. This is a pr. Well, it's not the pr. It's actually a push on the this repo on the carry The id carry resolver. Okay, so I published. Yeah, I published like Hello war showing How a carry can use as a alternative to pure d id Great and then there's a utility That has all the other functions using the the carry by using python great is do um Do you want to give us kind of uh I got your high level. What's your high level takeaway is as you're going through This I I can I can show the okay. Let me run the the script and I can go Okay. Yeah, that would be awesome Yeah, let me stop sharing Stop sharing You see the Yeah, looks good Okay, so if I run I'm gonna just run this Then then I can go through the code, but the idea is this one is using the same sigma Did come library But instead of using pr d id I plug in the carry the id so This is a hello war where Alice created the id and you see this is the carry the id how it looks looks like this did carry and this This identifier right Uh, but this this in order to resolve this identifier in carry you need to to go and talk to the asian or we need to come up with with a with An alternative and the way we we found that can be implemented easily. It's like adding Take this as a d id url another Uh query parameter that is the key event with this long base 64 Path, right And this basic 64 path You can decode it and actually get the inception event of the carry so in in that In that event you're going to find all the extra information that Uh That you need as as an alternative to d the pier that is the encryption keys The sign increase and the service endpoint Can you and and yeah, can you can you compare that to how pier did works? I mean it sounds familiar Right. Yeah, so so I compare this with the length. I write if you have a d the pier here this is a like similar d id That I plug in just to compare the length. So this is a d the pier a full d the pier and then you have this this one as a Carry d id that includes that include the the key event, right? so Once you have the I did A pier d id like this one You go and call a library and the library can resolve that in the in the document that shows you the the The keys the encryption key the signing keys and the service endpoint Right in this case in did carry you can do the same Plugging in the library that is what what I did and Decrypt that part and resolve the key event and actually You can validate that this Key event that is encoded here is the one that So that's one on the fancy sink of party things off Carry that you can validate that this part is the one that produce this identifier So everything is in in the same d id is self-contained and you can resolve it And get the all the information in it and after that once you resolve it In the next messages, you don't need to use the full d id I can just pass you this one the short version Because you have you Yeah, because you already resolve it And and that's gonna be really short and that's I think it was the main reason why daniel Asked for something like that because these peers were really really long, right? Once you start using this DPR in the id these com messages You need to pass all this da for the Sender and the receiver and that's a lot of information flowing and flowing again So if you if you just passed the first time and the second time use the short one That's gonna be I think it's efficient So roto both parties then have their own copy of the key event log Yes, once once. Yeah, also. Okay. This is gonna be like we call it non-transferable Identifier carry identifier Because they cannot be rotated. So it's gonna be just like it here It's gonna have you you create that with all the information and then you cannot change it You cannot change the keys and not the information of the document Uh Same way as you as you using dpr I think is the algo number two. I think it's the one So you you cannot change it right And But okay, once you risk once you resolve the first time you store it if you have the A cache like a cache in your your asian then every time you see this short d id You can you know what to do, right? So you're saying that the Long form of the did carry has the ability to rotate But the short form of the did carry is like that Current whatever the current state. Oh, no. No. No. No. No. No just this one Is you you cannot rotate it. It's it's like a fix Uh, so like Good question. Yeah So is the idea that the long one you send it in the out-of-bound invitation and then all the other messages you send the short one, right? That's because the out-of-bound invitation in this case. Yeah, but for example in yeah in the out-of-bound You're gonna send this one But the other party and then when I reply to that they need to also send the long one Uh, he's long long one to two Alice, right? Right. So like the handshake like the first messages that we exchange There are slightly heavier, but then all the messages we exchange afterwards with the same Session keys quote-unquote would be like smaller because we can resolve those keys Right, right. And that's what yeah, that's the idea. So what this is what I did is Alice So if you go to the to the messages Alice Yeah You the Alice created the basic message Yeah, with the laundry Because it has the full the full state here you can decrypt it But then One once one are Replied to Alice they can use the short because Yes Right So like let's say if you wanted to rotate you would just send like a message with like a new long carry the id and then from one end and then the other one they can still Reply with their previous carry the id Yeah, you mean when I rotate the d ad that's what I was saying No, yeah, yeah Did did it come already have all the mechanisms to rotate the ad so There is a feel in the in the message that is called from prior from prior So you you you send a new message and you say my previous d ad was this one So you have a like a challenge Right, yeah So I think I don't maybe Lance you're asking that because you're trying to see if you can use in carry That can be rotated the I mean the where the keys can be rotated that was for your question Yeah, I mean my I guess In everything that I've learned about carry. I always hold on to the fact that the pre-rotation mechanism that carry talks about is By far the most core value that that carry provides and so I am surprised that we're talking about um That can't be rotated Yeah, right because okay. This is the first step Just to have something similar and alternative to pure the id just for that. Okay, right So this is what makes it carry light. Is this why we call it carry light? Yeah, that's what we call. Yeah Daniels calls at the beginning carry light and okay Don't want to call it. Oh, okay. Sure Because you use this one can be self-adversible. So you you need to be resolved in line and And because the did come already have a mechanism to rotate the id So you are not talking you are talking about the ephemeral Yeah Yeah, that's what we're to be alone like the Real id that you want to use to keep it for a long long period of time that need the keys to be Rotated that's another story. Okay, so for for that one There are other mechanisms or producing the did so the did is going to be always the same short one But the what do you include in the query parameters are going to be different in this case in For the dp alternative we include the the insertion event like the only event that this this a id or this The id is going to have right, but if you're going to rotate it Instead of adding the key event because we're going to be multiple events of the given log you are the An uvi so the uvi what it tells is still okay You have to go to that witness or whatever asian and ask for the full key event log Oh, that's gonna be a same option of like a Did carry along for And the third option is instead of giving this uvi you're gonna just send that And based on your on the ecosystem that you are Uh, you're gonna call like a super watcher that everybody knows that exists And unresolved at the id so for example if we in our case if we use A cardano and we have a super watcher that We know that All the identifiers Create it in in cardano can be resolved by this Watcher we can call call it and and ask for the full key event log But that's okay. That's was a lot of things That is not needed for did it come right? Okay, got you so There is no connection between the ephemeral carry dids and the Persistent carry dids or am I right? Okay. Yeah. Yeah Same way did pier and whatever you're gonna be the yeah, right. Yeah, the dids is gonna be the same but the what you The needs of one or the other is different, right? in the simple simplistic Need for did it come you just need the inception event and you come Based all the section of any in the in the same In the same the id for and then remove it when once you You cash all the purpose Okay This is really cool Roto, would you be able to share this sample output in some way? Like could you check it in as into this repo as as Sample a sample run Yeah, I think yeah, it's already I copy paste You get the Did I copy lance did I copy because I'm sharing the screen I cannot find the Where's the chat? Oh the chat. Yeah, you have to go up. Yeah. Yeah hit the chat. Yeah. Yeah on the chat On the chat, I paste the repo Where this demo is included? Yes, so you can go and just run I'm I'm just wondering for those of us who who are not python programmers But would like to look at this Could you could you add a sample output file to the repo? Oh, yeah. Yes. Okay. Yeah, sure Yeah, I can also add it like a read me or something like that. Yeah. Yeah. Thank you. This this is great Yeah, that is great roto. Thanks for walking us through that Okay Awesome, uh, let's see. Didn't I have any more questions on that since we're on it? I mean certainly Our next bullet point for status is Uh, trust over IP trust spanning protocol And so obviously carry has been A hot topic and obviously did come also has been discussed and so The better we can understand how these things will Can work together. I think the better that we can understand how They fit in the conversation for the trust spanning Protocol and beyond, you know, obviously there's multiple layers to that. So It's really good Anything else on the carry stuff? Okay Okay. Yeah, great. Okay. So I guess I can share again and Have one quick question So I guess I can the in the data resolver How do you store the dids if you have like cash dids? Is that a particular to carry question or you mean that in general? No, like in did come like right like, how do you know That you can use a cash did instead of a long form? Oh How I we saw that in the code is like I try to find if you have the If you receive a short version you you see that is storing your memory Or in your decent whatever I mean, how you are out, right? If you know how the cash version you're out, you're right Don't know. Okay. I never mind that was Is that that's just an agent specific thing? That's not really a did come specific thing or How do you do that? Yeah, like like the agent implementation side of things Yeah, I guess like whenever like you receive an item Yeah, there's a there's a problem if you receive Like supposed that you catch is no longer valid product or whatever you I think you are out and Probably you need to send something to the other party Yeah, I don't know how yeah, but you don't even know where to send it to right because your cash is out. That's uh, yeah Right Don't don't lose you need Yeah, call my phone. Yeah Yeah, yeah, okay. Yeah, that's okay fair enough. So yeah, like I guess like it adds this extra like edge case per se Uh, but yeah, that's the only thing But yeah, I like it because I could decrease the size of the message And like you usually don't rotate beads that often. So I think it's uh, it's a fair trade-off Yeah, can we talk about um, kind of the practical Uh point there that you're making Alex Because I haven't done a ton of uh, you know real world did come How how how long do we think a conversation?  Is that yeah, I mean like I would use them like an interaction right like Whatever like a session is like an interactions like I think of them as like Session bids almost So if we if we take this zoom session that we're talking about would we have all negotiated new ephemeral dids To start this meeting Yeah, yeah, we can Uh, I guess Yeah, I mean like best practice is right like you're supposed to rotate keys and do all that kind of stuff So like our agents should automatically do it for us So like we shouldn't have like the same meeting with the same keys until we get hacked We need to create a new one, right? We should like create a new one like every single time So yeah, the regular rotation for every new session Uh is yeah a protective mechanism for us, but there is um overhead to it Yeah, like similar to like tls session like at some point like you need to refresh your Your authentication token and like you need to do like a new key exchange and like Uh like reset the the connection Yeah So maybe would it make sense for this type of dids to have like an Like an expiration time for the cache like hey like Indeed indeed it didn't come already resolved that that's that's a great thing of did it come So you have a way to rotate the d id you you have a similar d id that Ephemeral means that you you discard it if you have compromise or whatever you discard it and you have a way to To include I say this is my new d id And the d id can can be from a different method right and change everything So supposed that you are using a d id from one method and later you do that. Okay. This method is no no longer Yeah, I'm no longer traffic methods. I want to change this one because it's more Okay, I don't know more secure.  Oh, actually actually in no indeed it come you you did that in band So that's that's one of the features of did come So so the initial the initial setup of a session Or we'll call it an interaction It requires some out of band, but once you've done that you're saying that The there is a mechanism in the spec here for rotating The actual did instead of rotating keys Right. Yeah, and that's what one of them one of the main yeah features of You do Yes, like you will send like you're all did and you're new did and like Did come will handle like the crypto car like you're like make sure that like you sign you have control your keys and do all that Yeah, interesting So you can do key rotation technically with doing did rotation if your right is just Right, it would be kind of the thing That's correct. Yeah You know Okay This is great. This shouldn't be new to me, but I guess it is I've heard it before but Yeah, and the way it works is you you send When when you want to rotate the your DID you send a new message with your your new DID in the message But also at this field that is called from prior that you this is a is a field that has the Is a shawty Token that includes the signature of the old DID so the receiver can check that you are the owner of the old DID and validate That you have included everything So then you you you The receiver say, okay, this this guy's now is using this new DID And you're ready, but he's the same the same people the same person That's that's great. Thank you Lance. Would you be kind enough to put this tabs url in the chat? Absolutely, and I'm with you 100 bruce essentially You know I've got no ego on this and it feels like oh, I should have definitely known this But whenever I heard did rotation or key rotation my mind just said Oh, these are the same things like I you know, I have no reason to To to go look into this and yeah really I totally should Well, you can technically rotate dids and keep the same keys Oh really because like you Put like the same keys on Two different like store like two different storage mechanisms So like right like if you want to like store them somewhere else You can have the same keys and have like different method resolvers to to fetch those keys Now would that still work with a carry self-certifying? Did Yeah, I mean like it's just uh, they're just keys like in the cal those you have just keys in there So as long as you have you can store those public keys and like uh, right like on on chain Or something like that And put them there too like an ipfs some for example, I don't know Okay, interesting. Yeah, I'll have to connect those concepts in my mind better Because the so the the self-certifying part verse did peer Or did key Is there can we expand on that a little bit? Again, it feels like something I should know but uh, I'm fuzzy and if if we want to move on we can but uh Roto Yeah, I'm not sure like I'm personally like knowledgeable enough to speak on like self-certifying identifiers yet, so like I don't quite understand like I mean Yeah, like what you're asking for say. So, yeah, well, so so roto had said that Oh, you know, here's one of the nice mechanisms of carry That it is, uh, you know self-certifying uh identifier Uh, which you know, if I if I think about the diagrams that I've seen for carry, right? It's this strong binding between keys and identifiers and controller But then When I now that we're in my mind, I'm doing a comparison between did peer And carry I think it reveals that I don't Really have a strong sense of what it means that it's That that binding is strong like it does did peer have that same strong um three way right Yeah, that's right. Because once you receive a peer DID You can decode everything And get all the public keys in there, right? So the the identifier is itself the PID document So that's what we call. Yeah I think of it like you don't need the did resolver like if The identifier has all of the data you need to resolve it in itself without like a did resolver per se So like you need to transform the identifier to do something to get the data out of it That's my understanding Okay, so, um for so I think that what sam has said is essentially Dids that require Mechanisms outside of the identifier. Uh, there's a weakness there then right that that For instance, I guess he would say probably a did web or something like that if if if now that domain is hacked You need to rely on the resolver Yeah That's where the problem comes in but with did uh with did peer and this did carry That that rotos describing you don't have to rely on a resolver, right? It's We're saying that it's encoded in It's pretty much base 64 encoded in there. So like it's nothing else to do. Okay And then the the okay, and then the difference we've talked about before with did key is that the Did key is the same as the peer but only allows you to have a public Sign there's there's no service endpoint in the did document. Yeah, I'm not I'm not encryption. Yeah, fair. Just one one single one key For signing and that's it Yeah, it's like a pamphile. I think it is like a pamphile. It's just another way to express a public key like another format Like a JW key. We have expressed a key format Well, when we say though that you can uh Okay, so I for a second there. I was thinking, okay, I'm I'm strong between these three things and you know, how they work but then Now that you're saying one key Okay, yes Fair because you are saying that the identifier could be tie or you could have Yeah, I was thinking that it's the service endpoint that is the difference Or why did key uh, it's not useful with did come Maybe this isn't the best form for me to work these things out in my brain, but uh Anyways, I don't mind speaking out loud. So I we can I'll I'll chew on it for a little bit But uh, yeah, that's really good Yeah, I'm I'm in the same state uh lance. I think except I'm not quite as as as verbal What you're saying? Yeah, thanks Bruce All right, cool. Well, um, all right. So, uh, trust over IP. I think daniel presents this week Uh, I think he presents his proposal. Uh, there was a a lot of back and forth between daniel and um And sam let's get a link to that because that was nice and valuable as well Um That was Here Just need to Yeah, nice long, uh Discussion assuming I have the same I'm just going to scroll. I know that's annoying, but uh, yeah Yeah, yeah, yeah, this is the nice long discussion that I recommend. I'll post the link Here if anybody's following the Trust spanning protocol Stuff and uh, it's nice to learn from that and try to One of the things that I love about it is right, which it's this kind of first principles, uh debate of, you know Well, what is what are the first principles that we need to focus on for that layer and in a lot of ways that's helping me to kind of Separate the this the the incredible complexities that come with did come and carry and all these things and you know trying to uh It's helping me to to separate those into their constituent parts, so That's going to be that's going to be interesting I really appreciated the link to um to the to the video presentation That you put in last week That was Fascinating thing I thought so too and then were you did you uh, were you at the meeting on wednesday or have you listened to the the kind of follow on? No, I haven't if you have a link to that that would be super too Yeah, I'm not sure I attended the meeting so I hadn't looked for a link for it, but uh, yeah It's got to exist. So if I if I come across it, definitely Okay, good bang forth between sam Daniel and others Okay, and then Daniel presents his proposal this week at the wednesday TSP Something task force. Yeah, I think okay Good All right. So we did talk a bit about did come Okay cashing and those things. Uh, yeah, this did come v1. Um Um So sam did uh supply a link From that's just on did come org, which you know, again, that's one of those things where it's like, oh, why didn't I think to look there? Um, but here let me post it here Um Introductory, where is this thing? It was on did kind. Oh, I think it's on the guidebook Yeah Oh, here we go Yeah, now I know I couldn't find it because even now even though I know it existed It wasn't so easy for me to find so maybe that says something about the uh guidebook needing some updates But I will post that On here as well Okay, so that's a simple explanation of the benefits of upgrading from did come v1 to v2 Um, it's it's probably worth walking through that real quick. Uh, so let me pull that down here Okay, so, um So the version of did come in the aries community was version one and this uh spec describes the new version Uh, and the changes between them Um Well, how are we doing on time? Oh, we're almost out of time before instead of me going through this I mean we can but uh, is there anything else that we need to cover in the next six minutes or even less Because I have another meeting. I got to jump into Just just want to make sure we're not missing anything. I missed the um the aries, uh, wednesday working groups So I'm not sure how much they did on the aip3 it was a short meeting because, um Sam Sam Curran what had another uh had something come up and Stephen Curran Ah moderated the meeting Being unprepared. It was it was still an interesting meeting Yeah, a non creds, right because I think I sent that to ridolpho since he's done a bunch of non creds work Right, didn't steven talk a bunch about uh, non creds, especially revocation. Yeah, he's yeah, he took about yeah Yeah, just yeah the time frame from when asking for revocation to spend a lot of time on that, I think Which I'll admit I don't Really, I mean I've been listening kind of but I've always been working while they've been talking about the time frame stuff and So I don't pretend to understand it at all But uh someday Okay, so uh, yeah, then I assume there hasn't been much movement on that and we talked about the test harness stuff And possible tags for these things and then yeah essentially the tsp is kind of related to the grand unified theory Okay, anything else that Uh, we need to highlight or should highlight No, this is not Okay Good, so then let's quickly I think go through this Okay summary of changes formalization of methods used in v1 at jwm envelope ecdh1pu standardized form of auth crypt I'm trying to get decent at understanding the Nuance between some of these crypto things um But yeah, if anybody wants to say anything in terms of of these two things, I mean, we know that there's uh Uh, there was a change in the aries, uh envelope, um A spec specification, uh, so I think there's two separate rfcs for the different envelopes Uh Practical changes All right, did exchange is not needed anymore. We've seen lots of talk about this Uh, you know as like hakan and others are trying to get Uh implementations going This is kind of a big deal because they have designed in, you know, some form of connection Uh that requires did exchange and It doesn't work that way in v2 anymore. I think each message essentially Um Can stand on its own, but I don't know much more than that. Anybody want to see anything about that? I mean like if your next change did like brodo said Now it's easier. You just got to like create this new message with like the sign from with the from prior and That's how like you do your did exchange So it's kind of like built in the spec Like the ability to rotate this Okay, I hadn't I hadn't connected that that's um, how you start as well as how you rotate So maybe that's extra useful Yeah, like an idea workflow like you would like create like an outbound with a did peer And then you would switch to a data like you like feel like more secure towards right like that's kind of what we're talking earlier Yeah, it's like it didn't matter. It's more secure to do like your actual like issuance or whatever Yeah, okay cool Anyways, maybe maybe it's would be good to uh, I meant to read through this and I haven't so maybe it'd be good to read through it And we can talk about it. Um next week It sounds like a lot of it is uh Or it looks like a lot of it's related to things that we've touched on But maybe we could dive a little deeper so we can uh I don't know grow at the open wallet foundation like in the presentation that At least sam share with me like he's gonna talk a little bit about like exactly this difference between the company one and b2 so great Talk to them on on that meeting Okay, there was one other thing that I wanted to share um that maybe we can talk about I meant for us to get to it today, but That's okay uh, so in this Uh, I'll just pull it down here and just show it to everybody and I'll post the link so that people can read up on it uh I came across this comparison Of dwn versus, you know, uh did come uh verse carry and again. This is all being prompted I think by the trust banning Uh protocol talk, so I think it's all good But this felt like a very awkward comparison to me And maybe that's just how it's supposed to be because there's only uh, you know A certain amount of overlap between these But uh, all that to say if anyone's interested in seeing something, you know reading through that I'll post that there and maybe that's something that we can talk about Um Next week would would you guys like to discuss that next week? Sure Okay Do we think it's worthwhile And uh did come Okay Roto do you think that that's worth bringing up in the uh meeting this uh, the the user group meeting this uh afternoon Not to put you on the spot. It's okay if Yeah, sure Okay Which one is it? It's the this hack md where they talk about dwn versus did come versus carry I think Yeah, yeah Yeah, I don't know how much they've discussed the uh, the tsp, but uh, yeah, there's a lot of question marks on that Okay. Yeah. Yeah fair enough Okay, we're all learning we're on a journey here. Okay great to have everybody. Thank you for coming and uh, yeah We'll meet again next week and you know, maybe I'll see you in the other meetings today Thank you. Okay. Bye


Don't be embarrassed. Zumba ain't only for women. I'm telling you, besides August football practice in high school, Zumba is the hardest. Next to the football practice in August, Zumba is the hardest thing I've ever done in my life and still is. Every day, every day that I go, it's something new. Another challenge. I was 307 when I started. I should be 234.8 right now, so that gives me overall 72.2.


Teal low was pop We are on twitch. We are Not live, but you can leave a like comment subscribe turn on your post notification bells Let's continue to grow the family for Chicago to the UK right behind me. This is the Spotify playlist You got to be rated a seven or better to be considered to make this list So with that being said, you know, we doing music reactions This is rim tricks. It's called It's called did he party that's insane talk to me I can see the signs. You don't need to warn me that man a dodgy like a diddy party Sadly, we've got sweets. I mean it's martyrs. I wish I lived my life really smartly But I'm hopeless rolling with my nigga on my nose or never sniff at Charlie I might get locked up the fence really want one. I just ride out low like I'm in a Harley I'm happy until I see man. I'm winning calmly try change my life It's really hard some people have died and it really hurts me I'd like an innocent guy when I'm with my own fam. Don't know what I'm like when I'm with my dog He's hope I don't get a bird for my kids a As they shouldn't your family should never know what you about unless they in there with you Calls and have it but if not, you know, keep it kosher. Oh shit's happened.  It has no time and it's shit's great I had to say again for all the trips and got no choice God all the sticks from the hijack gang are all the whips orders throw up their bands and all the hip And my men said that he chased me but it was a other way round. I don't need to run Don't need to run My men said that he chased me but it was a other way round I ain't scared to don't know I didn't cut real cheese when I'm there in London We just want to make peace until the sun done created The real G that is what my mom done. We don't party like diddy. That is where the fun done And nowadays the hoods all fade down there. Everyone's naked. There's no me round Yeah, or the fakes are winning. I can't take around it.  I just find where they are when they're rearing up Smoke man like it's a garden flea. I've been shot stabbed and rocked and beaten up I'm still rolling to the wheels. I'm surprised.  Amen boss. No DD when I said I can't play with us Nearly was out. I really was but then I got shot now. I'm feeling stop shop Nowadays man will snake you from behind bed dodgy moves like dilly-dots. I'm still on the case I don't do it Charlie. I'm a straight yard man. Like see you mom. I can see the signs You don't need to warn me that man a dodgy like a diddy party when it's time to slide out bro bring the car All right, I got bored. I'm gonna keep it real with you like The most creative part of the song was the ditty bars But on the scale of one to ten one being average. I mean one being stop rapping five being average ten being The best thing I've heard just like a five. It's average This average it ain't gonna get you, you know, I'm saying a million views or on the charts or nothing, but you know It's cool Might have to leave it off my playlist though. I'm being honest here man Just my honest opinion just remember it's just my opinion man


The Narcissist's favorite word. This is a word that they will use when they are in a very bad situation, where their options and ability to act are limited and constrained, when they have no way out. It defines how the relationship functions within the world that they've created for you. You might have heard them say this word before, or maybe they just implied it. In every interaction or conversation you have with them, you will feel like this is what they're thinking of you, especially when there's an argument or disagreement. This is a word that they are deeply afraid of. They don't want anyone to see them in this way, which is why they will be so quick to use this word against you, because deep down they know that what they do falls under the words definition. The Narcissist's favorite word is crazy. They may call you crazy, or they will call someone else crazy. It's something they will use in a hopeless situation, when they feel vulnerable or at risk of being exposed. When they feel like something undesirable will occur, when you question or confront the Narcissist on something they've said or done, something that they know is not good, something that they know would make them look bad, they have to imply that you are crazy, and they may even say that you are.  It has to be because someone is having a nervous breakdown. There has to be something wrong with them. Of course they don't really believe that. They just use it to explain and justify everything they've done to you. And they won't always say that word. They might just imply it, or they might call you a Narcissist or a psychopath. They will use it whenever it suits them, whenever they feel like they are under attack, whenever they feel like they are at fault.  The Narcissist's favorite word is crazy. They always want to make you feel like you're crazy for questioning or confronting them. You're crazy for doubting them. You're crazy for having a different opinion or perspective, a different idea or preference. Even if they don't actually say the word, they always imply it. They indicate it by suggestion, if not by explicit reference. If they're in a bad situation and they don't have any ways out. If they feel trapped or cornered, they will suggest that you are crazy. Regardless of whatever logical conversation you are trying to have with them, regardless of the facts, they have to stay in denial to protect themselves.  They refuse to acknowledge an unacceptable truth or motion. They refuse to admit it into consciousness because they don't want to deal with the shame. They don't want to deal with the consequences of their actions. So they try to shift the blame onto you. They try to make you think there's something wrong with you. When there's a problem or difficulty, they see it as though someone has to be to blame. Someone caused this event to take place. Someone is responsible.  All they care about is their feelings and they don't want to deal with the feelings that come along with accepting that they are a fault. This is something that every narcissist will do. They won't always come out and say it, but they will always imply it. They will always imply that you are crazy. They will always imply that something is wrong with you because you have a different opinion or perspective. You have an idea or preference that is unfavorable to them. They want you to feel like who you are as a person is bad or wrong. They want to deny or invalidate your qualities, beliefs, personality or appearance.  That's where they're going to target because that's how they're going to control you when they are applying that you are crazy. It's not always something they're doing consciously. It can often be instinctual. It can be a part of a fixed pattern of behavior.  It's something they do automatically in order to survive a dangerous or unpleasant situation. The narcissist has to see it as though you're crazy. Something is wrong with everyone else all of the time. This is how narcissists feel. There's something wrong with everything in the world because they expect everything to cater to them. Everything has to work around them. Everything has to be what they want it to be so that they can be happy but they never get what they want because it's unreasonable. It's beyond the limits of acceptability and fairness because it's always something that you have to do. You always have to make them happy to make things perfect. It's like if only they had that one thing they're missing then suddenly everything would be great but they never have what they want. They never get it. So that's why they're unhappy. That's why there's always a problem. And the problem is something that they're going to take advantage of to get attention, to get supply. And while the problem could be a fault or mistake that you have made, they can also create the problem. Or they can push and provoke you until you produce the problem so they're behind all of it. They're the cause of it but they're not responsible for it. That is the most favorable position for them to be in where they can be behind the scenes causing all of this chaos and destruction without taking responsibility for any of it. They create a problem that makes their process easier. It makes something more likely to happen which then facilitates whatever they want to believe and validates their illusion. But no one can see that they are behind it. No one can see that they are responsible. So in their minds they're not responsible for it and then it just continues and sustains itself for as long as they need it to. Which is usually until one of you dies.  Why are they so quick to imply something like that about you? And the reason for that is because that's what they fear the most. They fear being called crazy. They fear being seen as strange or foolish because that's exactly what they are. They are crazy and at some level they are aware of it but they're trying to battle it out with you. They have to see it as though whatever behavior they're engaged in or whatever emotions they feel it's the result or effect of something that you're doing. You are the cause of it. It's something that you are responsible for and you are supposed to fix it because in their minds they don't have any control over it. They can't make a choice of decision about it. They think it's your problem. They think it's your responsibility which is why they will do whatever it takes to make you feel responsible. They have these feelings that they choose not to deal with. They express it in a way where you are forced to feel like it belongs to you. Where you are forced to feel like you are responsible for it and you accept it. You take responsibility for it. Which then takes the weight off of their shoulders and then you are left to resolve it. It becomes your problem which really is the definition of crazy and deep down they know this. They know that they are the ones who are mentally and emotionally unstable but they will push and provoke you until you begin to feel that way so that it relieves those feelings within themselves because when you accept what they're throwing at you it relieves them of the burden. It relieves them of their troubles and worries. All they are really doing is dodging responsibility for their actions. They are trying to avoid reflecting on their emotions and they are looking for an escape. They are looking for a distraction from their miserable reality. They are looking for a distraction from how they really feel about themselves. Thank you for watching. I hope this video has it with you. Please like, comment, share and subscribe. If you would like to donate, my PayPal link is in the video description. Coaching Choirs, you can email me at coachingatnarksviver.uk Thank you for watching and I'll talk to you soon.


Hello everyone, I'm your host Sarah R. Welcome to the new season of the Wikimedia Tech Talks, which are backed by popular demand. Tech Talks are an opportunity for individuals from the technical community to share what they know. Going forward, we'll host in-depth talks on technical topics about once a month. There will also be other opportunities for sharing in the future, including shorter lightning talks and demos. We'll share more information soon. Tech Talks are open to everyone, so if you have an idea for one, please feel free to contact me either by email or on IRC, and both of these things are in the slide in front of you, or follow the instructions on the Metawiki link that's also on this slide. Today's speaker is Subbu Sastri. He's the principal software engineer on the contributors team at WMF, and he will be speaking on the long and winding road to making Parsoid the default MediaWiki parser. There will be some time for questions at the end, and you can ask these in IRC on the YouTube stream, or if you're in the San Francisco office, please come up to the mic and ask your question. And without further ado, I will let Subbu take it. Thank you, Sarah. Let's see. Okay, do you see this? Yes. All right. Okay. Good morning, everyone, and my name is Subbu, and I'm going to talk about making Parsoid the default parser for MediaWiki. And so on the parsing team, our mission is three-fold. It's about Wikitext, the output of the parser, which is HTML, and about the parser itself. And today, I'm going to focus primarily on the last piece. That is, how do we use the same parser for reads as well as edits? And that makes the question, what exactly are the two parsers, right? So that's going to be as part of the background in the first part of this talk. I'm going to tell you what the two parsers are, why we have two parsers, how they are different, and what is the core reason why we have two parsers today. And as part of that, I'm going to talk a little bit about some details about HTML, the DOM, Wikitext, and how Wikitext behaves, and how Parsoid tackles this. And in the second part, once we have a background, it's easier to look at the roadmap in terms of where we started and where we are today, and why we are porting Parsoid to PHP, and how do we get to the end of this road? So first things first, what are the two parsers? So right now, if you look at MediaWiki, there is the default parser which takes Wikitext and outputs HTML. And there's also Parsoid, which not only can take Wikitext and generate HTML, but it can also go the other direction. It can take the HTML produced by clients like visual editor and produce Wikitext. So the default PHP parser has been in MediaWiki since about 2003. It's currently used for all the desktop previews. It's used by mobile web. It's used by the iOS app. And if you're a client that uses the Action API, then that's also backed by this default PHP parser. So when did Parsoid come into the picture? So Parsoid came in around 2012. It started as a project to support visual editing. But since then, it's come to support a lot of other products and tools. So it's used by flow, content translation. The Android app is backed by Parsoid output. It's used by the 2017 Wikitext editor. We'll enter extension, use Parsoid, and Google Qwix Offline Reader under STPI. The thing to note is that right now Parsoid is written in Node.js and runs as a separate service. So what does it mean to unify the two parsers? Right? So as it turns out for a bunch of reasons, all the routes essentially lead to making Parsoid that single parser for media Wiki. So first of all, the current legacy PHP parser cannot support Parsoid clients. And most and also importantly, so the output from Parsoid, the HTML that we produce, is annotated with information about the source page that is about Wikitext. And this information is useful for clients for extracting information about the page that was parsed. And long-term, it's really not tenable to have two parsers and it really hamstrings future work. For example, if we want to evolve Wikitext, we want to fix templates, then if we have these two parsers, then we have to make those changes in both and that's really not efficient and really slows things down. So this is something that we've been moving since about 2015, and which is why I'm calling it a long time. Long and winding road, because it also goes back all the way to 2011. And here I've actually played you the Beatles song, but I think it may be the case that if I played it, YouTube may take it down for copyright violation that according to the tech talk and we don't want that. But you can imagine the song in your head and how it kind of applies to Parsoid and media Wiki. Okay, so on to the next slide. So here are some pretty pictures of some long and winding roads and they are somewhat metaphors for the work that has gone on and that we need to do.  And occasionally it's a nice and clean path. Sometimes you see it looks like it's the end of the road, but you don't know what's beyond the horizon and it may even be a pretty picture. Occasionally it's uphill. So those are all metaphors for the project of Parsoid and where we are going broadly. So let's set the stage for the Parsoid unification with Aladdin's genie. So you might know the story. So Aladdin had been stuck in a cave, he found a lamp, he rubbed it and out came a genie and the genie gave him wishes. So let's say we have the genie with us and we could make a wish, right? So we could ask the genie, please port media Wiki to Node.js and integrate Parsoid with that and no services is going to be a monolithic codebase. So the question is, now that we have all the code for media Wiki, Parsoid, all in Node.js, how many parsers will we need today to support our products? Well, you may think we will probably be, just need one parser, but you're going to be wrong. We'll need two parsers today, right? How about we go the other direction? Report Parsoid to PHP, once again integrate this with core, no services, right? We have a single codebase. All code is in PHP, how many parsers will we need today? Once again, you'll need two parsers. All right, so let's make a different wish. We asked the genie, all right, port Parsoid to PHP, but also fix Parsoid's output and feature differences and fix extensions that need to be fixed. Then if you ask the question, how many parsers we need, then it's easy to answer, we just need one, and that would be Parsoid written in PHP. Or you could go the other direction, you could ask the genie, port some subset of media Wiki core and some subset of extensions in Node.js, and once again fix Parsoid's output and feature differences including extensions. And if you did that once again, you'll probably only need a single parser, in this case, Parsoid written in JavaScript. So this might be a little confusing, right? So the point of those slides is to kind of get to the core of why we have two parsers today. It's not because one is written in PHP, another is in Node.js. It's not because of the service architecture, whether we have services or whether it's a monolithic codebase. Of course, they do matter, but the core reason and the primary reason why we have two parsers today is because the way they process Wiki text is different. One is based on processing strings and generating strings and the other is that's Parsoid, uses a structured representation internally. It uses Stokens, it uses Saddam. And so as it turns out, this difference in processing and the difference in pipeline introduces a bunch of differences. And so yes, language and service architecture do matter, but more as constraints on the parser. So they affect the quistiness and length of the road but are not really the primary reason why we have two parsers today. So to make a little bit more sense of that, what I said there, right? Let's go through, let's understand a little bit about HTML and DOM. So HTML and DOM are web standards. And so if you look at HTML document, it has structure under semantics. And this is defined by the web standards. And so DOM is a document object model. It means when you parse a page, the written HTML and build the DOM, then you have a structure and the objects on the DOM have a relationship with each other and it's all defined by the spec. The reason this is important is because you can use HTML and DOM libraries written any language. It could be Ruby, PHP, JavaScript, Java, and they all behave the same way. So if you're a client that is looking at the output of Wikipedia, output of a page, then you can analyze the page and extract information from it. You don't have to deal with Wikipedia potentially and which is where we'll come to in a little while. So as an example, let's look at this piece of HTML and for that, this is how a DOM might look. So a DOM is a tree structure. If you use the regular trees and this is an inverted tree where the root is at the top. So there's the HTML element which has two children, the head and body, head as a single child in this case, the title of the document. In this example, the body has two children, the H1 and 8 acts. And the web standard and semantics of the spec define what this all means. And so if there are more children of the body, there would be more elements in this, in the DOM. And so given this DOM structure, you can query and manipulate the document. So I can, for example, ask, give me all the headings in the DOM, in this document. Or you can ask, give me the second section. Or you could say, give me the first column in the table with the ID, badminton, give me the first column in the table with the ID, badminton, tournaments. Or if you are interested in modifying the DOM, you can try to replace the first list item in a list which has the title, which has the ID, greater span, with the Beatles. You can do this with any DOM library in any language. So the question is, if you look, so given our parsers, we now know that the output has a structure and you can use libraries to query and manipulate the DOM. So the question is what about the input document? What kind of structure does the input have? And more importantly, can you map the input and the output? The reason this matters is because if you're trying to answer questions about a wiki page or you want to manipulate the wiki page, it's important to note that HTML is not the canonical representation of a wiki page. It's wiki text. Wiki text is the authoritative representation currently. So you cannot really answer queries or manipulate the page by looking at HTML and HTML libraries without having a mapping between the input and output. So for example, if you wanted to ask, give me the info box content and all its parameters, you won't be able to do that. If you say give me all the citations that come from templates, no, you can't do that. Or if you want to replace this info box with something else, you could if you passed and manipulate wiki text. But that means every single product, every single client that needs to do this would have to deal with wiki text and that's not really a scalable and good solution overall. So the question again to go back to what I asked earlier is what kind of structure does wiki text have? So do wiki text constructs behave like dawn trees? No, they don't. Wiki text does not have a spec formal semantics. So the behavior of wiki text emerges out of what the legacy PHP parser does. And that parser does not really think in terms of dawn concept at all. It's primarily concerned about very efficiently constructing the output HTML string. And this makes sense. I mean this was written in 2003 and at that time there was no foundation. It was all volunteers and more importantly the servers were not as powerful as we do now. And it was very important to do this very efficiently. And that particular design makes good sense. But that's not the case today. And so given this templates pretty much inherited the same behavior from wiki text. And so the lot of templates are once again primarily about constructing the output HTML string. And any meaning that you might think templates have emerges from conventions and practices and wikis are not because there's a spec or something that's well defined there. So to make this a little bit more concrete let's look at an example. Right. So here is a piece of wiki text. And so you have these triple quotes which are which signify bolding. And so in there you have foo and bar and you have a template and outside you have bars. So foo will be bolded clearly so the question is is bar going to be bolded. Right. So that's a trick question. It depends. It will be bolded if your template is one of these three on the first line. So the one next template is just the echo template is just output parameters there's nothing special. But if you had the templates on the second line bar will not be bolded. So already it's clear that you cannot look at a piece of wiki text in isolation. It depends on what templates might do and what expand to. So let's let's go further. Let's say your template was is this one X with a string A which has both tags around it. You would think that A would be bolded. Right. But the question is will it be bolded when it's included in the page in that wiki text above. No it won't. Who will be bold bar will be bold but A will not be bold. So this particular example I've given here is not at all hypothetical. So recently when we are replacing tidy editors had to fix wiki text pages and they had to go changing wiki text to make sure that when tidy was replaced the rendering will not break. And editors on multiple wikis encountered something very similar to this. So italics would the behavior of italics aspects would change depending on the kind of template they were using. So you would expect I think going back so you're expecting looking at the template that bar would be bold A would be bold but in reality that's not true. This would be possible if wiki text had independent parting that is if you pass the top level page separately from the template and then you plug the output in the hole where the template is. So that requires independent parsing that requires dom scopes without non-local effects. So that's the state of affairs with wiki text today. So there are non-local effects in wiki text and it has a bunch of implications. It has implications for usability so for editors and humans it makes it a little hard to reason about consistently. So you just can't take a look at a string of wiki text and say what it's going to do. It makes it hard for tooling because if you're trying to manipulate a page then you got to deal with all these non-local effects. Thirdly it makes it hard for getting really high performance because you really cannot break the page into independent chunks and parse them independently and put it together. You cannot do that as you just saw. So the question is how does parsing deal with this? So clearly the output has DOM structure. So what parser tries to do is it assumes DOM structures wherever it's required to make this whole thing work and given this far input DOM and the real DOM in the output it tries to compute a mapping between the two. So for example it treats extension output as an independent document. So that's a DOM fragment technically. It looks at link content and treats it as an independent DOM fragment. It looks at figure captions as DOM fragments.  But templates you cannot do that and where templates don't really map to a DOM tree it tries to expand the range of the template so that it fits a DOM tree. So for example normally as we just saw this particular template invocation with A surrounded by bolt tags would map to a DOM tree but in that example it expands the range of the template behavior to include foo and bar which is not really part of the template. So this is an exercise you guys can try later on take this piece of wiki text add it to a page and try editing in visual editor. You'll see the behavior of editing. So basically parser tries to scope and bound the behavior of a template so you can edit it as a single unit. So the other example where parser deals with DOM semantics or DOM structures is when you're adding when it adds section wrappers around wiki text sections and so it has to handle scenarios where a wiki text section doesn't map to a DOM tree and there are a lot of cases like this on actual wiki pages. So here is a sales pitch for parser. So broadly parser has demonstrated two big ideas. One is that it is practical to go back and forth from wiki text and that's how visual editing is possible. That's how content translation is possible by doing HTML editing in those products and secondly it can actually map an output DOM node to the input wiki text string and as I said it assumes DOM structure as required to make this work and the reason this works is because for the most part wiki text on pages is do the right thing and templates for the most part there are a whole bunch of templates which do the right thing but it also there are also of course wiki text which has broken markup there are templates which don't fit this model and the parser has a lot of hacks and fallbacks to handle scenarios where DOM semantics don't apply. So the other part of the sales pitch is parser it deals with wiki text so you don't have to and so the implication is twofold. So if you are a client that is concerned about reading the page and analyzing the page and extracting information from the page then you can use a spec that parser it publishes to query the output DOM and extract information about the input document. So for example Google used to run and query media wiki to get extract information for a small extract it doesn't do that anymore it uses parser it's output because it's much simpler to do that and if you're a client like visual editor or content translation and you're concerned about editing the document then you can do that entirely on the DOM you don't have to deal with wiki text at all and then you can rely on parser to map it back to wiki text faithfully. So that is the background so we talked about what the two parsers are and why we have two parsers and I went through some details of HTML DOM wiki text and what the parser approaches here. So in the next part of this talk let's look at the roadmap where have what is the history of parser and where we are today and evolution of parser itself and I'm going to also talk about why we are putting parser to PHP now you might have got a glimpse of it early on but I'm going to talk a little bit more and finally what is left to actually get to the end of the road so here is 10,000 foot view so I'm going to first this is again in three pieces so in 2011 and 2012 which is when parser was conceived so feasibility was entirely unknown for example when I joined the foundation in 2012 May 2012 a couple of people I know talked to me and were pretty sympathetic and told me that I'm joining a project which probably doesn't have much chance of success and it was in fact unclear where this was going and whether we can actually get this done to map this to a dump so in the period 2013 to 2015 it is a period where parser got established and it was clear it was here to stay and so in 2015 so we had this realization oh no we have two parsers now and also two parsers and two of other things and so 2015 onwards we started looking at unification and I'm going to talk about what that all meant so let's rewind to 2011 so this is from a page on Wiki where it says Wikimedia Engineering's key future-facing priority for 2011 and 2012 is to create a rich text editing environment backed by a revamped normalized and more consistent WikiText parser so that's pretty ambitious and so in 2011 May 2011 during the Berlin hackathon a new parser was planned along with WikiText next which I guess we are calling WikiText 2.0 and visual editor work on this new parser started around November 2011 as part of the visual editor project and sometime in February 2012 it was called it was renamed to Parside the thing to note here is even back then structured representation was at the core the plan was to maybe build an abstract syntax tree of WikiText and also maybe use the DOM and so in early 2012 the core design of Parside was in place and this is still the design we used in 2019 so you will take WikiText parse it to tokens you transform the tokens build a DOM and then transform the DOM and the output would also have annotations to expose information about WikiText for example what kind of link is it a Wiki link is it an external link is it a language link is it inter-wiki link for example and it also annotates information about extensions it demarcates extension output it provides information about templates the name of the template what's the boundary of the template what are the parameters of the template and other meta information for example so the plan in early 2012 was to prototype this in Node.js and then port this to C++ making this a PHP extension in core this was meant to be self-contained with fallbacks to core functionality only for extensions but by the end of 2012 this had changed a little bit so we had to do a lot of rapid development for the first release in December 2012 by then we had dropped the ID of a C++ port and we deployed the Node.js implementation and that's how we got a parsing service and we also dropped the pre-processed that was in Parside it had a whole bunch of edge cases because the processing model was different and it was not clear how it was going to perform so we relied on the MediaWiki API for expanding templates of WikiText so overall this particular design was working fairly well and we still had a lot of work to do we had a bunch of compatibility things to resolve before we could even call Parside feasible so it didn't make any sense to put in a whole bunch of additional work before this was done so we just talked to this architecture so of course this led to all the subsequent debates about services, third party support and so on but back in the back at that time it made perfect sense so in 2013 and 2015 support for VE had stabilized and new clients were appearing so Flow, Content Translation Mobile Content Service which is behind Android App Google was using this Kvix is using this and with the GSAP project we also built a prototype of WikiText Linting Tool which we fleshed out later during tidy replacement and we will talk about that later in this period we also had to start dealing with the ramifications of a different WikiText pipeline and a different processing model and we also had to deal with the fact that WikiText as a syntax was not really designed for going from HTML to WikiText so we had to deal with a whole bunch of issues around escaping WikiText and editors becoming unhappy about all the no Wiki annotations and trying to introduce a lot of heuristics to reduce those and of course on different Wiki editors were used to different kinds of templating, template formatting and so we had to deal with template formatting issues and we had to deal with dirty disks now the reason all these things matter dirty disks escaping formatting is because once again the canonical representation of a page that editors are used to is WikiText and a lot of the workflows that editors and admins have really advanced editors is entirely based on WikiText and it's based on WikiText source disks so it makes sense WikiText disks if you add dirty disks then the make they work really hard so anyway so I think all this makes the work of Part 5 harder and in this period we also started helping third-party users with their parcel installations and provided a Debian package for Part 5 it was also in this time that the debates around services heated up and Part 5 became posted child for the debate so I'm going to talk a little bit more about what I meant about ramifications of a different WikiText model we looked at this a little bit before so we have templates that produce strings that have no equivalent templates that produce just an opening tag of a div there's no way to represent that in a DOM you either have text or you have the entire content of a div you have the opening tag, closing tag and the content of the div you cannot just take a div and represent that in the DOM so for example there are info box styling templates that produce a style of the info box a part of a tag so once again you cannot represent that in a DOM you have table cells style templates which emit something else and there are templates which produce table start tags and tags and so this means we have to deal with template boundaries while respecting DOM structures so we looked at this earlier with that example and so rectangles in this case show the boundaries that Part 5 creates for templates and we also started realizing that we need Part 5 specific versions for extensions so we have site, gallery poem and a bunch of others and we also had to deal with bad markup and we had to round up it back without dirtiness I mentioned this earlier because editors would be unhappy if we did that and I'm not going to go into the details here but we can if this comes up in Q&A one of the most prominent markup errors is what is called faster content and this is when content is embedded in tables at the wrong place and this is HTML5 spec thing and as you know on Wikis templates are very common sorry tables are very common and so faster content errors are also fairly common and this has been a source of a fair bunch of complexity so now that Part 5 has been established we started thinking about what does it mean to go to a single Part 5 how do we use Part 5 output for both read and edits and also in this time we had to deal with the fact that Part 5 is a service and there is interaction between Part 5 and Media Wiki and so Part 5 has been making a lot of API requests to Media Wiki so we built an extension the Part 5 batch API extension to reduce the request volume and we also had to deal with timeout issues because of cascades between services we tried cascades on timeouts so this was the time where we were stabilizing Part 5 in terms of performance deployment in Q&A so we had about eight Part 5 incidents between 2013 and 2015 four of them in 2015 and as part of the work we did in 2015 we could stabilize it sufficiently that there was only one so this brought us to start thinking about the future so how do we start using Part 5 for reads as well as edits so we started charting options for how to identify the parsers we started cataloging all the ways that Part 5 was incompatible with the PHP parser and the first step here that we identified was to upgrade code HTML5 by replacing HTML4 tidy because Part 5 output was patient HTML5 and which was the source of a bunch of differences in the output and we also started respectively planning around Wicked X 2.0 so we started writing RFCs and proposals for fixing templates fixing Wicked X semantics and in retrospect this was probably a bit premature given that we had two parsers and we really wouldn't be doing this on both parsers but at least we are well prepared to do this okay so what does it mean more specifically to identify the two parsers so we have taken a two pronged approach in some cases we are going to move the PHP parser towards Part 5 and in other cases we are going to move Part 5 towards the PHP parser for example as I mentioned earlier the PHP parser output used tidy which was HTML4 based and so we started thinking of migrating that to use HTML5 and we also have a project which is not completed yet to migrate the legacy parser output media output to parser style semantic markup this is work in progress which will be completed later in other cases we started bridging gaps in parserized output which is like language variant support was missing which is once again work in progress it's mostly done at this point that linking was missing we fixed that so I'm not going to talk about all these differences I'm going to focus primarily on this one big project that we did which is replacing tidy so this was meant to be a one year project when we started this in mid 2015 but this turned from one year to three years it was a blog post we published last year we can take a look at the blog post for details so we had to do a whole bunch of things to replace tidy so first of all after a bunch of different attempts we finally settled on Remix HTML which is a pure PHP HTML5 parser and we knew we knew right going into this project that this is going to impact pages on wikis but we really did not know how that was what kind of impact we are talking about so we built a lot of custom QA tools to figure out the impact on wikis so first of all we had to spin up two virtual machines in labs one would run tidy the other would run Remix and we populated both this VMs with 60,000 pages from about 40 wikis and we revamped some of our QA tools we had for parseride and to we had to run a lot of of a page on these two wikis and then compare the screenshots and generate a diff but the problem we ran into is that these diffs had a lot of noise for example if there was even a one pixel vertical shift then it would render it will show a lot of noisy diff which was pretty much useless from the point of view of a new tool called upright diff which is based on video motion detection detection tool and this was used for doing a diff this would actually hide vertical motions which played a big which was very important in letting us get numerically quantitative numbers which you could then use for figuring out the impact of differences between tidy and Remix so once we had all these tools we then did a bunch of testing and then this revealed that replacing tidy would potentially cause a lot of disruption on wikis and more importantly we realized that editors might have to fix pages and templates to actually mitigate this impact the reason this is so is because wikis had over time come to depend on tidy specific behavior this very inadvertently it's not something very deliberate it just editors write wiki text and it behaved right and they just went on with their work but so to deal with this we initially started adding some compatibility code around tidy to handle some issues but it also was going to require a whole bunch of fixes to wiki text and templates so we had to build tools to assist editors first of all we got to figure out what pages were going to be impacted number one and given a page that was going to be impacted precisely what piece of wiki text on the page would have to be fixed that's something that we wanted to help editors identify so we built this linting tool that I talked about earlier which could precisely identify what was going to be fixed for that and we built a linting extension to expose this to editors and all wikis and we also built a parcel migration extension so that editors could actually make a fix and verify that the fix that the change to wiki text would actually fix the page and we then undertook 18 months of community engagement 6 months of which was actually preparation for the engagement and so as part of this we had a bunch of information on wiki published help around linter pages and made regular announcements and gave wikis a one-year window and so progressively we did this so it was clear that we couldn't make the switch at all in one go so we started doing the switch progressively based on monitoring how wikis were doing with their fixes and we managed to get some early adopter wikis primarily Italian I think German British wikis to switch over from tidy to remix and based on that we discovered that there were more tidy related issues that had to be fixed so we again fixed code in parsoid and PHP parser and made a final switch around July 2018 and finally we removed all tidy support from media wiki in version 1.33 so one thing that we it was very clear from this effort is that we have a lot of legacy constraints so here are some very prescient thoughts from the 2011 hackathon so Tim Starling writes there are a lot of features that people count down that rely on the current regime indeed and Neil K another person in the hackathon says a simpler parser seems possible but it becomes impossibly distant when insist on not breaking anything of course it's clear that there are a lot of pages in the process of upgradation that would really not be acceptable and as we see that a lot of work has gone in parsoid and in replacing tidy so that we can actually slowly make progress in evolving our technical stack without breaking the pages that we already have and I think if we want to do more of this in the future that is fixed wiki text that we use for replacing tidy as a template we can use parsoids wiki text linking abilities and use the extensions we have linter and parser migration migration to assist editors to fix pages were required and will of course require community engagement here so this is the only slide on the services debate so till about 2016 I mentioned earlier that will it have services or will it have no services and till about 2016 there was really no clarity around this where media wiki architecture was going to go more practically for us what it meant is that the constraints around unification of the parser were very unclear so in 2017 all this came to a head there were more debates there were position papers there were conversations working groups and summits in 2018 and the parser unification was one of the components of this platform evolution and one of the first steps was porting parser to PHP so let's talk about why why this actually makes sense porting parser to PHP because it's going to add more work we are trying to unify parsers and now we are going about porting now I think it would make sense if we look at what we talked about earlier so let's bring back the genie here so there were two wishes we made to the genie where we ended up with a single parser which one involved porting parser to PHP and which two involved porting a subset of media wiki core and extensions to Node.js in both cases we still have to fix parser whether PHP or Node.js output and feature differences and here is the reason why which one makes sense and the underlying assumption here is that we got to fix the service architecture of parser first of all it makes sense because it's a limited scope all you need to do is port the parser code based on PHP versus an unknown scope where we say port a subset of media wiki core and extensions to Node.js because we don't know what else the second important reason why it makes sense is when we identify feature gaps we can actually leverage core code to bridge those gaps for example internationalization and localization support for extension is missing in parser right now and we can just use core code when we port this to PHP there are some other side effect benefits from this non wiki media wiki is get a simpler install in some cases and we're going to get a potentially simpler code base in parser there is no async code there is no code to talk to media wiki and as we realize realizing we are expanding the exposure to parser code base to others during this port and we hope eventually I think there will be little bit more clarity about services once parser exists in the services dimension okay so once we port to PHP how do we get to the end of the road here are some milestones first thing is to finish the port and take parser out of service and no pun intended there and we think it's probably going to take four to six calendar months we'll see what happens there and finish some of the other projects like fixing the media output fix the already known bugs finish implementing language variant support identify any other parser feature gaps and bridge with core code into the internationalization support most importantly we have to fix parser performance so in order to actually map the input and output and enable clients to work on the DOM, parser does a lot more work than the current PHP parser and is going to be slower there's really no way around it right now so this will require some serious performance work including even potentially throwing hardware at and then we've got to focus on making parser the default and wikimedia wikis so which means we have to establish regular QA runs to identify any and core tissues just like we compared tidy and remix here we're going to compare parser output and the PHP parser output maybe run it once a week and see what we find analyze the results file parser identify if there is anything that editors need to do on wikis and we got to finalize a new parser hooks API and migrate over wikimedia extensions so I think the ad space is probably 18 to 24 months from now but it could be sooner depending on how things progress and depending on how many people get involved here since I think we are probably close to end of time I'm going to skip this but broadly I think the reason why we need to change is because we need to change the types of parser hooks that we currently have which are based on the PHP parser internals and finally I think once we are done with this we can focus on media wiki by publishing the new parser hooks on API publishing docs around fixing extensions and following the application process and hopefully that will bring us to the end and hopefully that will be the end of time I think we are all involved at the people who are currently involved in the porting effort and the names in blue is the parsing team and the names in green Brian Gabriel and Trevor were the ones who were involved originally in taking this project off and the others would have been involved with it. Thank you Subu so feel free to ask questions on IRC. Subu if you put that slide up again it has the information on IRC which channel and then also you can ask on the YouTube stream or if anybody in the room has questions let me know. Do you see the slide? Yes. If you have any questions and seven more seconds then I will assume that we do not have any. If you don't have a question or if you have a question later after thinking about it do feel free to either ping Subu or you can also ask me and I can send the question along to him as well. Okay. We'll give everyone on the stream a minute to catch up. Subu if you have any questions on the IRC channel the slides will be made available and yes they will. They'll be made available after this talk and I'll send out an announcement where you can find them. And then we also have a question for you Subu is the rollout timeline 24 months? I mean I anticipated problem is going to be 24 months after the actual port and after that is when we can actually get on with the rest of the unification project. Yeah. So I think it'll probably be 24 months from now. And then we have another question for you. Would you recommend that editing tools or bots start using parsoid instead of wiki text? I definitely recommend looking out and checking out parsoid spec because it's going to be much simpler. You can just use HTML manipulation. So that's definitely my recommendation to check out and we can provide you more information and help if you require to switch to parsoid. I mean just to kind of give you a sense, right? So the library that bots use is called MWParser from hell and I don't know why it was named that. I think I assume it is probably because it's hard to pass wiki text presumably and so I think it's probably going to be simpler if you just used HTML in some ways. But again depends on what you're trying to do. Okay any other questions from IRC or in the room? Oh one more question. Will there be a formal spec of wiki text? Well, so I think one of the three goals of the parsing team was improve wiki text and make it easier to use parse and not make errors. So I think we do, we would like to publish some spec at some point but it would be hard while we have two parsers. I think once that will require making parser the default and deprecating certain behavior and fixing some common cases as undefined and I think we would like to but we may have at some point but it's not right away and definitely if we move towards wiki text to point it will have a spec in terms of behavior and all of that. Questions? I don't see any other questions. I do see that there is a comment that a formal spec of the HTML DOM more or less. I think Cisca on the chat in on the office channel. Thank you. Yeah, I think so we I mentioned earlier that I think if you're a tool you can just use the output of parser and that output is spec under version and so if you're actually only dealing with HTML then that should probably be sufficient for you as a tool writer and I think we also have because it's version we also have a protocol for how to upgrade the versions and it won't just break on you so there is a content negotiation protocol and so you can pass the version header and you can continue to use the old version till you get a chance to upgrade there is a spec for the HTML output from Parser. Okay, we'll give everybody just another minute to ask any questions if they have them. I'm sad I didn't get to play the Beatles. You'll have to play it for us the next time you visit. Yeah, you can take it up with Sony. Okay. Well, I think that's it for questions if you have think of something later and you want to reach out please do. We'll be happy to pass those along to Subu. Once again, thank you Subu so much for doing this first tech talk of this new season. We're so excited to get started and I'm really grateful to you for being a person who was willing to step up and do the first one and really help out with these. Thank you for organizing this.