Okay. We are recording. So networking, there's been a little activity on the spec repo, a minor adjustment to the NR requirements. There's a discussion on the a ping, which we can kind of like share vital information or at least let the other know that there's vital information that has changed while at the same time saying I'm still alive. So that's still a work in progress PR for discussion. We might talk about that a little bit today. And it has certainly come to my attention after a few conversations that aggregation strategy, although looks relatively straightforward and there's a lot of kind of education and things to think through to make sure you get this right. So I want to kind of talk about where everyone's at, talk about explicitly some of these problems and see if there's anything else that we want to make it into the stack, even if it's just guidance. Cool. But we will start with updates and roll protocol lab is here all we want to get started. Hey, yeah. Hey, everybody. I hope everybody is safe and sound and everybody's fine wherever they are. For us we've been working really hard on gossip sub in the last few weeks.  We are evolving the gossip sub protocol, although there are no substantial protocol level changes. In fact, gossip sub 1.0 and gossip sub 1.1, which is how we're calling this evolution. 1.1 is a set of security extensions. It's basically 1.0 and 1.1 compatible with each other. So it's perfectly feasible for implementations across different languages that are supporting different levels of the protocol to to cooperate and to work with one another. Basically, what has happened over the last few weeks is that we assembled a cross-functional team here at protocol labs between maybe to be the receiving networks lab who is represented here today by the videos which are disco Filecoin and test ground. This was an internal cross-functional team and basically what we did was enumerate and recent for a number of attacks and their feasibility and the cost of those attacks and the potential impact in gossip sub 1.0. We designed a number of mitigation mechanisms and basically we hired a goal implementation taking this one as a reference implementation by introducing some of those mechanisms. And they are things like peer scoring, gray listing, backups and other elements to secure the protocol. We'll be posting the links to the spec itself, which has already been merged to master. Just find the right place. Okay, there you go. So there's a link to the spec, which is a document in itself. So as I said, it's conceived as an extension to the baseline spec. And there's a PR that implements these changes on the reference implementation.  And I totally invite everybody on this call to go into the spec to read the spec to comment on the spec. And, and also on the PR itself. Yeah, okay, next steps. Basically, right now we have a team of three people that are dedicated entirely to testing the effect of our changes against the baseline of 1.0. So we are building, we're using test ground for this. And we're building a number of attacks and coding a number of attacks, modeling a number of attacks and testing them against the 1.0 implementation and the 1.1 implementation. Once the right moment comes will release all of this information and the numbers and the improvements to the public right now it's sensitive. And yeah, over the next few days. There is an internal red team here protocol labs that we've assembled that would be auditing the spec and the implementation. And I think that once that checkpoint is over that would be the safe moment through to basically throw these changes over the fence for other language implementations to adopt. So JVM, NEM, JES, Rust. And basically, you know, we are here at the disposal of everybody else to support this work to support the implementation, the upgrade work in whatever manner is needed, right, knowledge, funding, whatever so we can we can talk about that. Also in parallel, we're going to be looking at potentially hiring a an external audit firm as well. So open to suggestions from from you guys, if you have any. That's all from me. I think I extended myself to that. Awesome. Thanks, Ro. Would you estimate this is in terms of the development side two days of work a week or two weeks of work to get up to spec. To get up to spec. I think it shouldn't definitely shouldn't be given given the fact that so basically what we're doing is we have the reference implementation. In the next few days, we're going to be annotating enhancing and even refactoring some parts of the reference implementation to make to make them more delicate so that you know it's very easy to like go through and actually understand how things relate to the spec. There's a bunch of like algorithmic stuff here that needs to like very clearly be, you know, cross link to the spec itself. So hopefully would be super digestible. The spec and the reference implementation themselves the pair of those. I would expect magnitude of days. It's is not a huge. It's not a refactor is not a rewrite. It's a bunch of and if you implemented it with the right hooks which is the way that we've done it we've implemented it with a set of like callbacks and a number and very easily plug into the right places within gossips of itself to inform the score, which then can run kind of like in a decoupled fashion so yeah I would say magnitude of days. Cool. Thank you. Any questions for all. Thank you. Yeah, appreciate all the work there. Felix you have anything you want to add on every five or otherwise. Yeah, so the go implementation is now up as a pull request on go Ethereum repo. I've been. I've implemented a new LRU based session cache for it so it doesn't no longer stores keys in the on disk. And in general we've reviewed it with the go Ethereum team. I do need to make some further changes to reduce the time the unit test take because right now the unit test take about 60 seconds and previously discovery unit test took about 10 seconds. So I need to reduce the test time. And yeah, other than that, we are positive it can go in very soon. We have also identified a couple of things that should definitely go into the next spec version. So looking at the issues right now. Dimitri from the one of the Java teams I keep forgetting what this client is called. Sorry. Harmony is has proposed that we should change the definition of the find note packet to be. To not require so many individual requests. So we will convert it to have some kind of like to basically accept multiple distances in a single request. And doing that will basically reduce the number of requests required during a lookup. And we have also identified the problem with the tag construction that is in every packet. Because there is a type confusion there so nodes can under some circumstances misidentify a packet which they receive and this basically leads to a situation where you get a lot of ugly messages in the log. I mean there's basically there's no there's no big danger there is just something that is easily avoided if we change the format a little bit and we had some ideas and discuss it. So these are the two most pressing things I think for the next iteration of the spec would be to. Oh yeah and then there's one final thing that Yannick suggested which is to potentially use compressed compressed elliptic curve keys in the handshake. So this would save something like 32 bytes for every handshake in the in the packet size so I think it's worth doing but it's backwards in compatible change so it will have it will go in the next spec version. Yeah so I'm going to be busy updating my implementation with these things to see if everything's still working and I'll create the next spec version over. Yeah probably this week and and I'll notify everyone. And then we can just. Yeah. Keep improving. Not really. Thanks. And otherwise, from anyone else and either. For testing. Well, and others are talking earlier about more like the lower level testing. Meanwhile, Luxman and me are looking into this network testing somewhere between like your application level and this power level testing. So we have a rumor and bedroom. And those are evolving and the Python wrapper is getting very like integrated with the price pack that you can produce all these consensus messages to send to clients. From these network tests and then start some initial sync test group of concepts. So this is all being worked on. Luxman is new is helping us part time. And he has this repository up called status scope. If anyone is interested in helping, giving feedback to like proof of concept testing. For sync test, for example, then these results can discuss these posts. It's still early and I don't think I want to bother every client with very fast testing right now. Cool. Thanks for it. Pro is also now using rumor, pyroom and a modified version of the Python spec to pull down to sync the lighthouse test. Pretty cool. So within a little bit less than 10 minutes. I think nine minutes something. I think it's still disabled. The spec can sync the 10,000 spots of my dust does not. It's kind of fun to check that it's actually works. And you can load your own configuration and this person 0.1021. So I think everyone on that version can try it out for themselves and sync that doesn't take a transition to match. It's not really useful to have some network doing to be able to interact if you're about that also can produce content is messages rather easy. So if you want to scripture on tests, scripture on how monitoring or something, you can do that. Other updates will move on. Okay, so as I mentioned at the beginning. Challenges and edge cases in the aggregation strategy and every subnet. I think it's been on a few teams minds. Specifically, speaking with age, he wanted to make sure that everyone was on the same page and that we were kind of hitting making sure that you're hitting all these edge cases. To that end, I will give the floor to age, if he wants to kind of discuss his experience as far and some of the issues that he's running to age. Yeah, sure. I'm not sure how far all the teams are into implementing that naive attestation aggregation strategy. But we've, we've hit a few issues I've talked with various teams about so maybe other people have some answers or kind of made progress in each of these areas so might be a good chance to have a chat about them. I guess, so some of the main things that we've had to think about is discovery we need discovery to be relatively quickly relatively fast. So, we've spent a bit of time trying to, I guess tweak our disc, our disc v5 implementation so that we can find peers quickly. We need to find peers because when we obviously when we need to subscribe to particular subnets we need to have peers to be able to subscribe to. We've spent a bit of time looking at the validated client to beacon node API, because with the new spec updates there's, you know, there's obviously changes in that API, specifically around getting the validated client to know who's an aggregator as well as the beacon node to know who is an aggregator so we've made some changes there so I'm curious to know if any other teams have actually modified this API to see because we've kind of gone off on our own little tangent and potentially we can still perform there and maybe some people have some good design kind of decisions there. There's difficulty we had, well, I guess it's not difficulty just kind of have to sit down and solve the problem that you have to deal with the timing quite, quite thoroughly with us being able to firstly discover peers and then subscribe to this particular particular subnets for a given future slot and committee. And if you have many validators attached to you kind of have to deal with that. One of the other issues is peer management. I've kind of been, this is one of the, one of the bigger issues I think that I've kind of been bringing up with everyone. When we have, when we need to be able to find peers to be able to connect to subnets we need to know. Firstly, the peers that we're connected to what long live subnets they can, they have this this involves knowing the ENR of the peers that we're connected to. So we don't actually have a mechanism for finding that out if a peer is connected to us through the live Peter P stack. So that's, that's one of the issues that we came across. Another one, I just, I don't want to just kind of throw all these things out there but another one that I've just recently kind of run into is that if we are an aggregator for a particular slot or subnet. So that if we're not an aggregator should we subscribe to the subnet. If we're not an aggregate I don't think we actually need to actually receive any of the attestations on the subnet we can just, we can just publish across the fan out. I'm curious. If we do do that then there's less people overall subscribe but we should have the backbone from the long live subnet. I guess that's just a small lift off the top of my head. Some of the things we've run into I'm curious to hear if anyone else is thought about these or run into any of these issues. For the API I can comment on that. So in the older spec is basically just one trip right value data sense the selection proof to the beacon node, and we can know verifies that this is We can know verifies that the validator is actually the aggregator at the right slot and then package all the session together broadcast the aggregated selection object by the latest back there's the in the latest back the aggregator has to sign over the aggregated object. So you so then it becomes a two way trip validator sense the selection proof we can know the response back with the aggregated object and then the validator signs it over and then send it back to the beacon node and then the big enough broadcast the object over to the net network at this. That's my experience so far. Sure, so if you guys have an API be interested to have a look at it. Some of the small I guess edge cases are that we're signing a future slot and this line is based on the signings based on a fork so in principle if you're signing an epoch in advance you could have a fork change. In between, in between actually having to do any kind of attestation and this to see the slot signature that you have. Another thing we ran into is that I think Marin brought this up is that potentially you want to give the option for different beacon nodes to actually do the the aggregation. So if you're connected to three beacon nodes you probably only want to use one of them to actually send the aggregation off so we we engineered I guess a way for when you do the subscription that the beacon node tells you whether you're an aggregator and not so that it's up to the validator client to actually ping the beacon. I don't want to get into the technicalities I guess probably better offline. But if you have an API to be good to have a look at and I can have a chat with you about how it's designed ours. That's good. I, I, I, I was sure that would be awesome. Cheers. So the other one which probably is just a quick one for Danny thoughts on if we're not an aggregator should be subscribed to the subnet. Apart from being long lived. That's a good question. So there's, there's There's certainly immediate value in there. So our subnet. If we don't do this have grown much larger than we expected them to because of the cross linking at every slot. So normally, you would have previously would have subscribed to a subnet, maybe an epoch advance, but you don't have a bunch of other people in similar or in similar slots, subscribing as well. So what this does is 164 of all approximately 164 of all validators are going to be subscribed to a subnet in the given time and so the subnets are much larger than we originally intended them to be. So if you're not an aggregator that's actually very nice because you can just send to the cult of fan outright. And not worry about actually receiving those messages and just the aggregators and the persistent. Those persistent will receive messages. The one downside that's worth considering is that validators don't then immediately kind of get weight to their fork choice through the individual attestations they would have seen. So there's a little bit of latency and a higher dependency on the aggregators to show up and do their work to add to that like a subnet committee fork choice, like group work choice. That's the persistent committee is going to be updating their fork choice, and the aggregates are going to be sent. The aggregates are randomly selected and they're going to be sent on a public channel very soon, you know within four seconds. So my intuition is I was actually a great change. I will think about a little bit more but open to other comments now. Actually, something that has been in the back of my mind is, you know, can we make these subscriptions more granular so that you're modifying the protocol such that you're only participating in a subnet on a particular slot but this actually, I think removes any of that complexity that I've been totally chewing on. So we defined by slot instead of epoch now, is that the plan. No, no, no. So the the idea is, if you are an aggregator, you actually join you subscribe to the subnet, so you receive messages from that subnet. If you're not an aggregator, instead you just find peers on that subnet and publish to them. So that if you're part of a committee, you generally just publish your message, the persistent committee is going to see it very quickly and the aggregators are going to see it very quickly, and then publish aggregates on the main on the main channel. If you're an aggregator you actually are going to be be seeing all those granular messages and for me to be so this saves a little bit on bandwidth, and also just reduces the size of these nets. Yeah, so one of the other problems that I mentioned is that we don't know the long lived bit fields or just a bit filled from the enough peers that connect to us. And I think we should have a mechanism that gets us that this is I mentioned this in the last networking call. Has anyone had any thoughts on this? I made a PR which has some comments on it, which introduces also a ping protocol, which is just a draft. I'm not sure if people have got thoughts about that. Yeah, I would agree with that. So if you don't have like any sort of time out, we know when to connect for how long to particular subnet.  So I think it helps a lot. In AGC, if only aggregators are joining subnet, you still there's, it's still worthwhile having the persistent, persistent subnets passed along because you can still make slightly better decisions on your peers, correct? Yeah, yeah, I think I think we need the persistent ones. So my intuition was reading the ping protocol is that instead of sending the NR sequence number, sending some sort of local identifier with respect to just your note in general. And the payload if that changes is the NR plus potentially other local information that might have changed. So it's kind of a meta sequence number. So that this ping protocol becomes useful beyond just the NR, if we want it to be. Yeah, sure. The small details I expect to kind of change and I'm happy to do some iterations on the PR is kind of mainly fishing for whether, firstly, do we agree, do we all agree that we kind of need all we want a ping RPC method and do we want to be able to like have a dedicated in our method or do you want to try and just take the bit field out and try and chuck in the status message or something like that. I think they're the two main kind of ways we could go. I mean in the ticket I argued for a separate NR message simply because it's a big blob of data and technically we don't need the NR to run or would it be nice if the two protocol remain separate from a hard requirement on NR even if it's kind of in the spec right now then it would be nice if you could run the rest of the protocol without the NR. It's really, they bring in peripheral stuff that is not really needed for it to itself. I guess this is something a bit of a design decision, right? I mean we were there last time in the last call. I'm not sure if it was the networking call or the general if two implementers call but we just did discuss it a little bit. This question of like should the note metadata be related to NR or something else and I think we just need to make a decision. Well there's two levels, right? So one thing is whether we relay some data over NR for the PR lookup and then there are like in protocol needs when you're already connected. Yeah, so I think that like for real time in protocol information exchange, NR is not really like the best format also because it always has to be signed. So it's something that like the signature in the NR is very important when the record is relayed because it protects the record in transit and also when it is stored in some other location. So it cannot be modified in transit but since if you are connected to a peer directly you already have an authenticated session with that peer. So any updates that peer is sending are first of all more recent than whatever it could be in the NR that you found earlier and also since it's authenticated you can trust the peer to tell you the right things about itself. So it's kind of okay not to sign this information. So I think the NR itself is nice but it's really more for relaying information about the note outside of the real time communication session that you have with the NR. That said in the discovery is a bit different of course but it's kind of like in general I agree with you that using NR as a way to like relay some application specific metadata in protocol is not the best way. I think it would be better to define some other protocol message for that and not have the signature and things like that. Gotcha. So there's two paths there as Asian point out the we have one thing that we want to we know we have one item that's persisted that we want to relay, which we can drop the application stuff that we can drop into the into the status message or assuming that there might be other semi persistent information that we want to really relay every once in a while we can do essentially a node sequence number that is dropped into a ping. That's just kind of bounced back and forth every once in a while. And if I notice it changes then I can request for the larger payload right now the larger payload would be the application subnet. It could also be other other information that was in the R or other mission in general. So one thing that I go in. So one thing that would be quite nice is to make this the format of whatever this message is to make that like, yeah, to basically define that format in a way that allows it to also be transported in through other means. So one thing that will definitely come to just be five in the near future, I think it will be optional is this idea of having the like pre connection negotiation. So kind of basically like a way to exchange arbitrary application specific information through the discovery network session. So this is not the same as doing, you know, like, you know, doing everything over the discovery but it's more like if you want to connect to a certain peer it's good sometimes to be able to figure out in real time if this connection is really going to be worth it. And this is what the pre connection negotiation is for. And if you have a way to encode additional metadata beyond what goes into the ER, then this format could also be ideal for the pre connection negotiation so I would really appreciate if it could be defined in a sort of like separate section of the spec to just have this like peer metadata object. Right, right. And this would also be an ideal place to relay things like the full list of like hashes for something or like list of gossip subtopics or whatever you need, I don't know like basically things too big to fit into a single unit. Got it. I guess another question. The, do we, we all want a ping protocol. Correct.  I think I need it because at the moment we don't have a way that if peers kind of just drop out to identify that shouldn't that be at the level. I think it's an advocate. Sorry. This point, I tuned out for a minute. So what I've seen anyway, in a rustle, there's a, there's an IPFS ping, which pretty much does a very similar ping to what we're suggesting here, which can check for peer liveness. We haven't specified any other little bit of protocols apart from apart from the RPC that we've built and gossips up, which have like these long lasting streams. So identifying when peers kind of drop out. I think we kind of need a ping protocol. So if the underlying transport is TCP, which is the case right now because we haven't really adopted quick. I'm not aware of implementations that have adopted quick yet. Then this should be part of the, so if a peer is connected to you, it means that the TCP connection is alive and we do run keep alive some of that connection anyway. So there are TCP keep alive that are happening. So basically when a peer drops out, you would be getting, if the rust implementation is behaving properly, you should be getting a disconnected notification immediately when the connection dies. Yeah, interesting. I have to check that. So one thing that I can say to that is that it's not always a good idea to rely on the TCP level keep alive because TCP keep alive is something that you cannot influence. And it might also get, you know, like there can be issues with that. So it's like the TCP keep alive is kind of a last resort option and you cannot really influence timing of it. So like you can, but it might not be respected. TCP keep alive can sometimes also fail and it's something that is much easier to do at the application level. So in the DevP2P transport protocol, the way we do it is basically that we do have this ping and pong messages which are sent if no other message is sent for 20 seconds or something. And it's basically, it's much easier to just put it in your application protocol because if you do it, first of all, you can use it to relay other information like the ENR sequence number or the metadata sequence number that you were talking about. But also basically it will give you, it will assure your note that basically the peer connection is indeed alive and you can have a basically authenticated liveness signal that is way, way stronger than just relying on sort of like the underlying internet infrastructure to do this for you. I would really recommend putting it into the application protocol because then you can really be sure that your keep alive mechanism is working and not mangled by some middle box or whatever. Yeah, I think that you do make a valid point Felix, like personally I'm not 100% sure of what the assurance is in terms of like, you know, the wider internet infrastructure in terms of the keep alive actually getting delivered. And what, so there are other layers and the B2P stack, particularly the multiplexer, the YAMX multiplexer does run pings and it is a lazy ping.  So it's only, it's only used when, when there is a period of inactivity that it, but this could be an even across multiplexers because I think that the Mplex multiplexer does not run keep alive. So yes, if you do see a benefit of using a ping payload at the application level to transmit in the future, like some other control message or some other metadata message or whatever could be useful. Yeah. But basically the two points are, I don't know about TCP keeper lives. In general, I think we have seen them work well, but it is also true that most of like IPFS, for example, and Filecoin do use YAMX, which also have that, that ping mechanism. So I'm not sure if it's just the keep alive or the combination of both that, you know, makes this mechanism work well for reporting connection, you know, connections that are killed immediately. So the thing about the TCP keeper life is that really the reason for TCP keeper life, why it exists in the first place is that usually when the connection is broken. Then they're like when you close the TCP socket in the in your user space code or if your process exits, the OS will take care of like sending a TCP level control message that will notify the other side that the connection is now over. And then basically the other side will in general receive that notification and it will basically relay it back into user space on the others on the other end of the connection. So you will basically see the connection drop. But the TCP keeper life is sort of the intention for TCP keeper life is to take care of the case where one side basically gets firewall off or just drops out. Because, you know, the battery runs out or whatever. And then in that case, no control message will be sent by the OS because the OS is dead. So it's kind of this is what the keeper life is for because if the keeper life extension is enabled on the TCP connection and it's something that is negotiated during TCP handshake. The both sides will be waiting for, you know, like any message or, you know, like this like keeper life notification, if nothing happens, then basically the connection is also considered dead. But the timeouts for this can be really long and it's all a bit messy. So if you do it in the application, it's just safer. And I think this this thing that you were describing is that having it doing this in the multiplexer is like a totally okay solution because it's really only needed to take care of this case where, you know, like the connection drops due to a power outage or something. Yeah, so are you are you guys using in Rust and the rest of the implementations are you are you guys using a Yamax or in books. We have Yamax and it falls back to Emplex. I've seen some interesting things on the test net, which is why I'm kind of proposing this way we just get like protocol negotiation timeouts. I have to probably have to track down these a little bit more detail as to the source of the problems. But I think in these examples I'm using Yamax and I still find some weird things so potentially it's not just the pink potentially I should see disconnects as opposed to what I'm actually seeing. And I'll look at that for the implementation specific. If I may like I also like, like to recommend an experiment with using the IPFS pink protocol. It's a protocol beyond just checking if the connection is still open. Actually, we'll use peer routing to find new addresses for the peer if this connect happened beforehand. And so for the case that was being highlighted like someone changing IP address because the changing location. From like regular ISP internet to some like for Geo spot because the the ISP went down or something. The IPFS pink protocol will ask the network again and like real real established a connection so it's beyond checking the connection open it actually checks if the peer is still alive and running. So so maybe yeah like checking that out and see if that like suffices for what you're looking for would be good. Like, I'm currently drawn to an application layer ping with a metadata sequence number and an RPC method for get metadata, which would be just longer persisted data like attestation subnet and things that we want to share like that is do people want to agree with exploring that. So as a template PR, we can have a discussion on or remove anyone. Cool. We'll take it there. Other further comments on this, and I will take a look at the IPFS pink protocol, which might serve. There's one thing like talking about long lived data. Some of the data that's been mentioned as long lived might change at forks. So if we specify something that's called a low message. We can't specify that it's just a low message like that it's only allowed to send on connection basically. I think that we'd want to just call this like a get metadata message and specify under which conditions you might ask for it, and even leading up to a fork, you might ask for it. Yeah, exactly. And like not required that the clients call it at a specific point in time either. Because if they need the data, they will naturally do so when they need it right so. No, like I think in the hello message it said something like you know you have to send it on connection or something. So something like this. Yeah. Probably not needed. And I know there's a little bit of desire to include like client information and version I don't want to include that in the core protocol but maybe in metadata we can leave a, like as a requirement, but maybe in metadata we can have 32 bytes, crap, somebody can just write to if they want to. Okay. Adrian, is there anything else you wanted to bring up with respect to so many other items like discovery subnets you mentioned that you're doing some things to quicken being able to find peers with certain things in their in ours is there. Anything worth sharing on that. I can just tell you kind of the some of the strategies that we're doing in case other people have built. Maybe people have already got this by default but so firstly we, we had a system where like an added peers could kind of get into the DHT. We remove that pretty much by taking a larger majority. Before you update like your in R. There was a there was a PR. I also kind of made to the, to the spec which means that our in ours don't have to specify an IP address so when you first create an IP when you first create an in R and you're not sure that you have an externally contactable address then you can kind of just leave the IP address out so it's easy to see that that your in is kind of either behind the net or hasn't found a way through the net. There's a thing we, we kind of borrowed from the P2P CAD which is in the queries the find no queries. So in, in discovery v five, there's, there's a time out on your on your UDP packets. Because UDP is lossy. We have the option to send packets multiple times after a particular time out. Originally in the one of the earlier versions of the implementation that the timeouts were, were kind of quite large. And when you're doing the query query has usually a parallel parallelism parameter, which allows you to, you know, send multiple queries, find no queries to peers at any given time. And for peers that were either disconnected or not not available on the DHT we kind of were waiting for this time out. Before we we classified the peers being non contactable and go into the next one for the query to kind of progress. So, instead we, we've added a, like a smaller timeout for like a, it's kind of like a query timeout so let's say that your, your UDP request has a timeout of four seconds. Then there's a query timeout of like one second or two seconds which after that period of time you, you don't consume your parallel parallelism parameter you keep asking more and more peers, but you still accept a response from other peers so that kind of speeds up some of the find queries. In terms of, in terms of looking for particularly in our fields, we decided that it's, I guess it's still implementation specific but in one of our earlier versions there's a particular, the, when you do a find node query to find peers you the query finishes and either you found the maximum number of peers or you've exhausted the, the crawl. In the particular case for subnets we probably want to have queries that finish early if we only need three peers or two peers or for example you can you can specify the query to end when it's found a certain number of peers. So we kind of filter the peer list during the query for in our fields as opposed to getting a chunk of peers finding out whether they match your in our field and then, you know, reprocessing the re redoing queries. So there, I guess some of the main strategies we're going to try that will hopefully speed up our queries to find peers for subnets. Any questions for age before we move on. Maybe I can just share one more thing which is can be like nice trick. So, the way I'm considering to do the, like in Goa theorem what the way we do it with the with the discovery is kind of like the design is basically like iterator based. So you can kind of just get something like a never ending iterator of like notes that you can pull from and then when you pull from this iterator it will just keep advancing the lookup. And one thing I'm considering to add very soon now is basically since the lookup implementation is event driven and I think the one in Rust is as well. It's kind of actually there's not a big cost to just having many lookups concurrently. So basically one thing you can just do is you can just run like 50 random lookups concurrently because they're all just state machines. And then if you want to limit the resources, you can just put a rate limiter on the number of outgoing packets or something per second. And doing that can give you a pretty broad scan of the network at a configurable rate, which is something that I can really recommend doing to like get many candidates as quickly as possible. And also I think that the, I hope everyone understood that. And the other thing is that I think that the change that was suggested by the harmony team to improve the find node to take multiple distances as the parameter will also speed up these queries because you won't will no longer have to send like up to three requests to find all the nodes that you might need. So the goal is really to like get find as many nodes as fast as possible like purpose of the protocol and basically anything we can put in the spec even to accelerate that is, and I'm super happy about more suggestions. Thanks. So, when you're working on this, consider some of these strategies to make discovery particular here's a particular in ours quicker. And this is in that API distance discussion on a updated protocol between valid or client deep node. I know that prismatic has their own set of APIs will certainly cross pollinate those. Make sure we're on using the best ideas. We're likely to do a ping with metadata and define some sort of metadata request. And then on the aggregators, my intuition is that are for non aggregators my intuition is that the non aggregators actually don't need to subscribe the subject that message publish. So I will work on those two potential PRs with respect to ping and redefining how you describe or not. And otherwise, I think it's mainly working through some of the details. I think some teams have worked in more of the details so please talk to each other. I'm sorry. Yeah, sorry you two. Okay, you go first. Okay, hopefully this is just really small. I got brought up in the in the agenda as well as that for beacon nodes that have valid is attached I think my current thinking is that for the peers that are connected to us we're going to prioritize connections with peers that have greater the larger number of long live subnets. So anyone have any concerns with with doing that. So these peers will bootstrap the subnet is it. So let's say we've got like 10, 10 or 15 peers connected to us. Five of them are just normal beaker nodes as in that they're not validating so therefore they don't really have any long live subnets attached to them. If I'm a validator, I want to find peers that have a long live subnet so that when I have to perform an action on one of those subnets. I've got a kind of a collection of peers that I know already kind of that subnet which saves me having to go and find more of them. So if I if I do go and find more and I have a peer limit or I only have a finite number of resources. I'm going to kick probably peers that don't that aren't subscribed to any subnets in favor of ones that are so that I don't have to do discoveries in the future. Sounds pretty pretty good to me. Yeah, it sounds like certainly the viable strategy as a validator I think the one concern, which I think was brought up was if you have these like super nodes that are running tons of validators and subscribed to all subnets at all times. They might become kind of naturally become central nodes in the network, which is something to consider. I think we definitely want you definitely want to have a diversity of here with respect to persistent subnets. I'm concerned about having finding peers that have the maximal number of subnets, even though I know that is kind of like the best way to optimize that problem. I also think it's kind of important not to be like have the peer set be too static simply for the reason that like if you get locked into a particular set of peers and you keep optimizing that, then it can lead to a situation where it's very hard to join this club because so it's also something that like but you can this is something like I guess in every implementation that you have to find like the good policy. Like if you like you only have so many slots available for connecting to other nodes so you might as well just you know like try to keep some of them available for you know like I don't know like more sort of like dynamic and experimental peer connection so you actually do notice if there is like new good peers on the network. So what my question was my question was that from from the dispute five implementers, how are people handling the caching of nodes. Do you guys in terms of the sessions not sessions but I mean the like discovered nodes so do I do are you keeping like persistent cash or in memory caches only or. Yeah, so we for us anyway we the discovery v5 invitation itself just puts them in the in the in the routing table and then lighthouse when it shuts down stores it on disk and loads it back up on this. But this is only for the for the table. That's right. Okay, so one thing that you can totally doing is something that you know is also like optimized for is that it's actually totally fine for you to keep an additional sort of cash on this of like a note records that you found because you can always fall back on those records in in the future I mean they might be stale but. It's something that is very, very valuable and go with your for us is like on on the ether one minute to have this like notes database where we basically store all of the notes we find and level to be. And basically means that when you start up you already have a pretty complete view of the entire network. Locally, and all you kind of really have to do is just you know like keep adding to that or keep re verifying those notes, you do kind of have to. Expire notes from this cash from time to time like you can just you know like keep accumulating note information forever because of turn, but it's definitely very helpful to like keep persistent caches off the network for a longer time. Because it will really, really help with rejoining the network, but also. Yeah, just it just means that if you need to do queries for like you know, like if you've already discovered a note on a particular subnet like half an hour ago, then probably if that note is really like interested in either it is still alive. So you might just, you know, connect back to it again. So if you just I don't know if you guys are using something like SQLite or whatever or level DB with a custom like index or something. You could totally just keep you know, like an on on disk index of like notes, which are on a particular subnet and and then just try those and you don't even need to hit the DHT for that you basically because you already found them. While ago. Yeah, it's a really good idea. We'll implement that. Thanks. Yeah, thanks. Any other questions for Felix or age or otherwise. This is a great way to find out which validators are on which people know. Yeah, that's true. Yeah, I had a question on about data privacy. So with currently how the subnets are like structured. It would be pretty easy to like map IP address to like a value this public key because because like the requirement is that each at the station is aggregated. So identifying like one value that was an IP would be easy across like a period of few walks. So is that something that we're expecting. It's certainly a known problem. And Secretly your election and other things on the horizon or to address some of these things and the decoupling of validator from node allows for validators to create more sophisticated setups where for example I might publish at the station to one node, whereas I only publish my blocks to another node, assuming I might get to us. So certainly a bit hand wavy. It is But very, very expected to this point. To be able to be an automizer network. Thanks. Moving on. Nicola. I took a look at your table. I don't know if I fully parsed it. Let's take a look at your table. Yeah, I sent into the chat. So it's something that we did. At it's easy just after last call with protocol on that Dimitri Micaela and others. And the idea was Let's be clear about the requirements so that everybody can agree that they meet this requirements. And so it's protocol on the menu that will propose this with the idea, for example, typically on privacy. Is it possible to the anonymize the network or not with the initial target that could be well, let's see that if it is actually possible and we expect that it will be possible with a plan so then for phase two we can decide on what to do. So there are just a few lines at the end of the day which it's about formalizing very high level things like how many nodes do we expect to have in phase zero and are we happy with 1,000 nodes and what's the target for phase two could be 20,000 nodes. So the idea of this is really can we find a place to have this kind of high level requirements in the specification and a way to say okay, are we happy with those requirements and do we think that they can actually be met by the implementation and protocols today. So it leads to two questions. I mean, do you feel that it's reasonable in the people in the call? I mean, I mean, there's requirements, 1,000 nodes, do you think that it can be done for phase zero? And are you okay with the approach? Should we put this somewhere in the specification? We can certainly take it to an issue for discussion and decide where we want it to live. My intuition is that we'll certainly have on the order of 600 to 2,000 nodes in early phase one. And that's certainly one of the most under tested items at this point is scaling nodes probably beyond the approximate 100 range. Yeah, and do you think that we should target 100 or do you think that we should target 1,000? Oh, certainly 1,000. 100 is the only thing we've seen, I think the maximum, approximate maximum we've seen on a test map, the prismatic test map before. So that's why I mentioned that number. I think 1,000 is probably in the range of what we'll see, but I wouldn't be surprised if we saw two or three times more than that in early days. And are people the one thing that we can achieve 3,000 nodes today? For example, aggregations or discovery, whatever. What's your intuition on that client team? I don't think anyone has seen bandwidth or finding peers or any of the networking components that we might be concerned about be the bottleneck or main failure points at this point, which is good. But we've also only have 100 known networks. So I think there's optimism, but a bit of an unknown quantity there. So do you think that we could put this somewhere in the specification? So there are known and then we can establish test plans around this and these kind of things. Yeah, why don't you port the table to an issue on the spec repo and we can go from there. Okay, I can do it. I actually drafted out sort of like a test spec on how we could sweep the different parameters to figure out sort of like given the current configuration, kind of where things start to break down. I can add a link. You all can look at it. It's basically like to find some stability and stress tests where you kind of sweep number of nodes. Validators per node, total number of validators, obviously total number of nodes. And I did, I guess. And we're also working on a test net plan on the types of things that we need to see in test net type of stresses, the number of nodes distribution of clients, that kind of stuff, which would also feed into validating some of these requirements. Anything else on this? Well, for the table. So point of the table also is to face the requirements, but just may not does not try and define steps in between. So that actually does make sense to work towards it. And but we do have action points. We do hit some limits. Anything else on this and anything else for Nicola. Okay. Next question. We hit this a little bit on item three. We definitely talked about the ping. We have a potential route forward and we're going to take conversation to that PR maybe if you are a little secretive. Are there other spec related items specifically a zero stuff that's going to affect us in the next week. We want to talk about modifications, questions, issues, common. Yeah, I had, I had some questions regarding sort of the, the assumptions of beacon nodes should, should kind of not on beacon notation. You're breaking up really bad. You haven't trouble understanding what you're saying because of the latency. Okay. Maybe it was just me stuttering. Okay. I can hear you now there. Okay. So, so basically I just had some questions regarding what the expectation is on how, on how beacon nodes are going to treat like light clients. Or other non beacon node. A couple of places in the spec where it mentions reputation. And I was curious specifically like, I would like client not to be able to implement all. So should beacon nodes going to disconnect from the node that doesn't, that doesn't implement all the. Are they going to, you know, are, are we going to say that if a, if a, if a client can't adhere to all the musts. Should you just connect and it's still, oh, look how we thought about like, like clients. Right. So I saw some of that discussion online and I think it's a valid discussion. The must certainly have not been thought about with respect to like clients in the past month as we define this spec. Are you still talking I still hear like some crunching. No, no. The latency keeps going in and out. So, I think it's worth doing a pass and see what is not viable. I imagine that a lot of the must might be with respect to some of these got some validation on some of these subnet. Exactly. On some of these subnets, especially the access station subnet, the light client doesn't need to be a part of them at all. So those are, those are out the window. The block, the block validation and how like clients are getting blocks. I think it's probably one of the biggest concerns as you brought up. And I don't have the answer. Yeah. So I was talking to Michael about this and would polynomial commitments be a solution to that. A zero knowledge proof would be. Yeah, but there's a solution to everything. In wave. Yeah. Yeah, but it was certainly a sophisticated there and all truth could tell you. Hey, this actually is valid with respect to things that you do know, but that's not necessarily the path that we're going to go. So I, you could certainly listen to these subnets and pick up blocks and just not forward them because you don't know if they're worth forwarding. You could also another path is actually not listening to these beacon blocks being broadcast in real time because you can actually do anything with them. The light client thing protocol actually would probably be a separate protocol where I'm talking to some sort of light client server. They provide me with the proof that advanced me from point A to point B, say, skipping a day's worth of or skipping an hour's worth of epochs or something. At which point I have a finalized route and I just get data about the state. I actually don't know if there's any value in a light client doing a block thing and actually getting all the blocks. Instead, they're kind of just skipping ahead. So the answer might be that a light client actually wouldn't participate in many of these protocols and is more just getting to advancing to checkpoints and querying state with respect to those checkpoints. That's kind of how the light client security protocols imagined today and I don't know, I don't know the value beyond that to actually be participating in some of these subnets. It actually would be nice because it reduces the bandwidth requirement and doesn't scatter light clients throughout your report gossip, at least for the core consensus here. Are we concerned about being able to tell the difference between a light client and like a regular beacon node? We're certainly concerned about the protocols that a node speaks with respect to what I want to do with them. And so if a node shows up and then can't handle all the things that I want that I might disconnect from them. Either via the ENR, via the advertisement of some sort of protocol or a component of a status message that I imagine once we do a light client thing protocol that information will be readily communicated such that you're pretty aware of who you're talking to. And if you're not a light client server, and then you're not serving these these proofs about how to advance to some point in the future, you very well might disconnect. And the light client very well might disconnect you from you as well because they see that you're not serving that protocol. And so you're not of any use to them. These are all things that are being chewed on but certainly have made their way into the spec. I think it's definitely worth doing a pass on the spec to see if there are any components that we estimate are going to be requisite for a light client that have any must that would make them non viable. And I will do that. Okay. Do you mind if I open up an issue that maybe we can just kind of. Yeah, cool, we can try to enumerate any problems there. Okay, cool. Thanks, Danny. Yeah, and thank you. I, it's, you know, these two, the designing of the networking spec and the consideration of light clients compared to one of. I haven't allowed those items to cross poly in my head. Yeah, totally. Other media thoughts on this item like clients issues with PPP spec. I mean, another like, sort of random thought is that if ultimately light clients and beacon if we can tell the difference between them. I guess in my head, which is probably flawed, I figured that, you know, the end state of the network there, there would be, you know, swarms of light clients. And that help kind of anonymize which nodes are actually validating nodes, you know, to kind of make it more difficult to find them. But, but I guess, I guess that's not really, this is sort of a problem already right now with each one right so it's sort of just businesses usual. Yeah, ideally we can find like my client service better. And that we start this thing sooner or later with a good protocol. The, in terms of swarms of light clients just kind of being embedded. Because they're almost certainly not going to take place in these kind of like consensus layer gossip protocols, they probably wouldn't be to embedded deep in there. Because the light client, but nodes in general, just a node that wants access to the network that is running a full node can and would be embedded in some of these consensus layer control air gossip topics. And if I'm just a full node and I'm not validating, I very well might not join any attestation subnet. And would instead just listen to, I might listen to aggregates attestation aggregates, I might just listen to blocks, because blocks are going to include a lot of information that I care about. And as long as there's a majority validators successfully participating it's very safe and I get the reduction of bandwidth. And the subnets might only be composed of primarily validators. Gotcha. Okay, thanks Danny. Okay, we're three minutes over any closing comments. Okay, meet stacked meetings this week. We'll talk a little bit about the v011 one release that needs to drop ASAP. And then we'll be re-release of a of our phase zero bug planning program and other fun stuff. Talk to you all then. Thanks everyone. Thanks everyone.


Well, that's my talk. I was volunteered into this, too. Apparently Tim thought it would be interesting to hear stories about people getting benefits from actually writing. And I kind of went, well, that's weird. I'm a software engineer, by the way. I don't write. I don't feel comfortable writing. Writing scares the living daylights out of me. So I thought I'd ask, who here considers themselves a writer? And by that, I mean somebody who can't help but write or enjoys it or thinks it's just normal. That's statistically speaking, in this room, we talk a whole stream for writers. And I think we don't quite get 30% as people who consider themselves writers. So yes, you are not normal. But obviously, as I was asking, I mean, we were asking before, there are more of you here who actually do write and consider writing as important. And that's one of the things that I fell into when I started my docker journey. I started sometime September last year where I did the weird thing for a software engineer and I read the manual. And this is how I then formed my opinion of the product or project and started using it. And it kind of snowballed from there. I found myself not being able to help but fix things. What is your opinion of the docker project? I love it. I cannot find a way of doing the things I do with docker that's better yet. But then again, I know that I work for docker. I'm not sure that I'm really looking for an alternative. So there's a little bit of guilt there. For all I know, Rocket or LXD might actually be more of what I want, except I don't think so, maybe. So I was kind of curious to know, those of you who aren't writers, why do you write? The knowledge that you gain has to be communicated and if no one else is going to do it, you've got to start somewhere. Do it yourself. So kind of the victim complex thing. It's got to be done. It's better than the Stockholm syndrome. Damn it. I'll do it. Yeah, exactly. GitHub complains if there's not a readme file. I have very, very many GitHub repositories, but I never noticed that because it's the first thing I... This is the point where I kind of think maybe I'm not a software engineer anymore. Yeah, yes. And so all of this aside, I now am a documentation maintainer for the Docker project and do this sort of thing for a while and somebody at Docker rings me up just after they get some funding and say, hey, would you like to get paid to do this job that you're doing anyway? And it's like, why would I say no? What strikes me after a year of working for Docker though is that there are almost no writers involved in the project, even now. And I was really... I mean, I don't understand because I don't know why writers choose to join the projects they do or even worse. I don't think I've actually joined a project that has a writer. So for me, I have a question for you guys. As writers, how do you pick your project and what can we do? Because we need writers because I'm not one. That's not a rhetorical question, right? No, seriously. You talk a lot about managers. Yes, yes, but that's... Even though there are people who are like that, they're the minority, the ones I'm actually interested in or knowing how to promote is the idea of people who aren't assigned a task but choose this project and write for it. Obviously, you open stack guys have got people who do it. I was going to talk more about projects that I have chosen to get involved with as opposed to... I mean, obviously, I got involved in open stack because I work for Rackspace. So, you know, I get paid to work on open stack, which is kind of cool. But I mean, I've also done stuff with... I've done stuff with little groups that I came across at LCA, actually. Even though the first LCA ever went to, one of the guys put up a slide at the end and said one of the things they were looking for in their funny little project that no one had heard of was documentation people. I walked up to the guy after and said, hi, I'm a technical writer, can I help? Because this sounds really cool. I've worked on LibreOffice mostly because I used it and it was a project that I cared about because I was a user of the project. So I think a lot of it comes from personal communication and just what you're working on. I've got spare time at the moment. I happen to be using LibreOffice. Hey, I'll contribute to the docs. So a lot of it is accidental, almost. Yeah, which is hard to attract. Although you do have a good point in that I don't think we've written much in the way of tutorials or anything to make it attractive to people who aren't technical to begin with. So, you know, the first thing I actually did with Docker was to create a public and Docker container because I needed public and for something I was trying out because of some job I was applying for. But, you know, so that would actually be more interesting. And that's now in Red Hat Stocks. But that hasn't led to anything either. But, yeah, that's a point. Obviously, writing documentation for the users you want to attract and then you'll get some proportion of them. And making sure you ask for that help as well. It's not just about having the documentation there but making it very clear that you are looking for documentation help. Yeah, asking for help is... Sometimes less willing to just jump in. Well, I... Yeah, and so the point was asking for help and that's true. I don't think we're actually actively on our page saying we need help. I also wonder whether that statistics, where there's just so many developers being interested in Docker, for example, whereas documentation people haven't got a reason to care yet because we haven't given it to them. So, cool. Ah, that's another problem. Yeah, I'm a bit more proactive, possibly because I'm a director rather than somebody who gets paid a regular wage. So, I tend to document stuff to encourage people to work on it. Yeah. So, because I know that if you've documented something, somebody else is more likely to pick it up and run with it. Indeed. And so, yeah, update the docs on it, turn it into a magazine article, spread it far and wide, tweet about it, whatever. Yeah. And you can actually get people going, ah, actually, I could work on that. Yeah, that's the thing that I thought was really interesting in the last year. It's like the hype around Docker is ginormous. So, it's not the spreading of the idea that seems to be missing. It's that other thing to convert documentation or writers into being interested or contributing. Because the hype around Docker is amazing. And so, you know, it surprised me that that hasn't converted into more writers coming along, just magically. Two minutes. I will write documentation for whoever pays me cold hard cash. Yes, indeed. Me, too. Yes, indeed, me, too, yes. But the thing is that I'm curious to know whether there are parallels. There's at least 50% of the contributors to the Docker project as encoders are doing so on a volunteer basis. And there's at least 20% of them who are not being paid cold hard cash. They're just curious, interested, and contributing. Now, to me, that should translate into some percentage of writers who are also going to be interested in doing this for their own... Well, I would add that as a programmer, I've learned that documentation is an integral part of programming. And basically, I document everything I do because for my clients, that is a service that they need to have, even if they don't know they need it. As a consultant, you learn it pretty quickly, yes. And even for myself, so I guess I would have expected that a lot of programmers would want to document their contributions to say Docker. Unfortunately, reality says not really. It's kind of the minority who really do want to document. There are those who then grudgingly do it, and then there's the ones who often forget pointing closely. One of the things for the company I work for, one of the things we've been asked for a lot and which hasn't really been mentioned at all in this whole thing is video documentation, where you actually have walkthroughs and that sort of thing. And that's a whole interesting, another question, and I was just wondering what your idea is. I mean, you see a lot of developers, but generally they're somebody giving a talk with slides on, but actually walkthroughs of how to do things. I was asked about that last week by some people, and I had to answer, I personally don't know, because I don't understand the medium. For me, watching a video is just too intrusive, time-wise, so I don't do it. And so I don't think I understand enough about it to actually produce some. So, I mean, there's an opportunity to actually find people like you who might be interested in that so that we can know what might work. And even better, you want to come help?  Commit five days a week. That's where I am. Indeed, indeed. All right, well, that has been very interesting for me. Hopefully you guys got something out of it as well. Thank you very much.