as you put it out. I think it's really an angular resolution, not a pixel resolution. So it depends on how far away you are. So I think we should go ahead and get started. So I think today we're probably missing some of the Vulcan CNCF people because it is their bi-weekly call. So starting from the top. So welcome to the next Network Service Mesh Meeting. We have three meetings. We have this one, which occurs every Tuesday at 8 AM Pacific time. NSM document call, which occurs weekly on Wednesday at 8 AM Pacific. And we have the NSM use case, which occurs every second, fourth, and fifth Monday at 8 AM Pacific. We do have a NSM use case meeting coming up this next Monday. We are also participating in the CNCF Telecom user group, which occurs every first and third Monday at 8 AM Pacific. Notice the trend here. And so that one just occurred. So it'll be not next week, but the week after for that one. There is also a CNCF networking group, which occurs every two weeks on Tuesday at 9 Pacific time. So we can remove KubeCon EU. We've already gone over that. It was fun, but now it's done. We'll have fond memories of KubeCon EU. Indeed. It was a great time. We have KubeCon China coming up. And we have an intro in the maintainer talk. And there is also the second telecom user group working group kickoff that is occurring as well. We have the dbtk user space, which occurs a week before ONS. I believe it's like the 18th and 19th of September in Bordeaux. So I've submitted a paper into that talking about similar to FIDO, the state of FIDO and NSM. I submitted a similar how NSM is using dbtk to further the aims of its users. So we will have, if accepted, then I will be giving a talk there. We have ONS Europe itself coming up. The call for paper ends in just under two weeks time on June 16th. We have MEF Los Angeles. Do we have anyone who's going to MEF representing there? So I had to cancel MEF because I have twins due right at that time. So I'd like to stay married. And I've canceled all late fall conferences. Congratulations. Congratulations to you. OK. Bitty good fortune from ToxicCubeCon North America the same week. We'll be able to congratulate you. If one of them's a girl, we highly recommend Ariadne, isn't he? Once again, I'm trying to stay married. So we have, so the question is, do we want to keep MEF 2019 on this? I think we should probably scratch it. There is still a small chance that they use NSM, but our product managers are kind of throwing everything in the kitchen sink in their demo. So I kind of just feel like at this point I'm going to triage and move on to other projects. So we have CubeCon coming up in November. The call for paper ends July 12. And that is going to be on from November November 18 through 21. There is a new to be announced edge computing world coming up as well. And so that one may be, I need to talk with the organizer to see exactly where he wants to take it. But if there is a networking component or networking path, which sounds like there is going to be, then we may want to see, we may want to work out whether this will be useful for us. This is started by the same guy who is doing IoT. We did IoT world. So if you have any other events, please add it to the website. And please add it to this list. I assume Lucina is not on. Oh, hello. I'm here. Oh, you are on. Cool. So social media community team, you're on. Yeah, CubeCon Europe was a really good event for folks learning more about network service mesh. There were call outs for NSM in the opening keynotes, as well as in the Telecom user group meeting and various other presentations during CubeCon EU. And I tried to search for anyone mentioning us by our Twitter handle, as well as NSM, network service mesh, all one word or three words, and tried to retweet as much as possible. So we gained 45 more followers over the course of the last three weeks. I slowed down with following folks, just over 50 new folks. And we've got a total of 133 tweets. And fun news. I actually went to a park in Barcelona where there was a labyrinth and statues of Ariadne. So I'll try to do some fun posts about that, as well. Oh, that's pretty cool. It was a beautiful space in Barcelona. And I think I've posted the recap video of the intro and deep dive. And if there are any other, oh, actually, I can also post a recap of the FTIO presentation, because those videos have been released, as well as the Linux Foundation networking video. So I'll get those out this week, too. Thank you. Oh, fantastic. You're welcome. You can see those go by. So, OK, so release notes. There haven't been any changes to the release notes. I was waiting for the branch to occur before rounding up Nikolai and Ed to try to work out any major call-outs that we want to have within the first set of release notes before finishing it off. I think there may be happy news on that front. Have we wrenched? I think the folks responsible should probably speak up. All they did was get up and look at the slack traffic. Are the folks responsible online? Yeah, more or less. OK, Nikolai, have we wrenched? Yes, we did. So, yeah, after having a little bit of crazy weekend, like it was failing, then in the end, we apparently merged two patches, which were literally one line each, mostly deleting code.  So we have the branch now. I still have to figure out how we are going to publish the images, because the credentials are somehow in the CI and probably need to figure out some automatic way of publishing the release images. But that's something to be figured out and to happen. I think that we're in a good situation, I mean, in a good position now. So, Nikolai, I understand looking at the CI as it's going. We're still having, and I've created a new tag for this class of things, things that I'm calling CI bugs, which is AWS just decided it's never giving us a cluster this time, which seems to happen remarkably frequently, or GOMOD is having a connection tip for some reason, which are actually not problems with our code at all. But it turns out, cloud provider is very wildly in their reliability in terms of providing managed Kubernetes. Not only that, we have seen, for example, our Kubernetes deployment on packet failing, because Cades.io was failing to provide us the relevant packages for QPADM to finish its work. So things like that. I mean, like. Also, don't forget that a lot of this infrastructure, not our infrastructure explicitly, but a lot of the Go infrastructure implicitly runs on, and Docker, runs on Google Cloud. Actually, Docker one doesn't. But. And so there may have been some issues that we may have had intermittently because of that. Something that we should consider, though, is there's a, we can spin up an Athena server in some of these locations that can help store some of this information, some of this context. So it may make sense to bring up a proxy and store some of the packages just to help with the GOMOD if that becomes an issue over time. Although it won't help with network connectivity out for the CI, so that's the only problem we run into. Unless we can work out what cloud they're running in in what region, but we probably won't get that visibility. One of the other things over the weekend with Google having issues is I saw a joke go by that's all too true, which is, it turns out when the Google Cloud goes down, so does almost everything else. As one person on the hacker news posted to claim to work at Google, Google's messaging service went out to that they used to describe to work on the headers. They claim to have backups online to deal with this problem, but yes, their backup system is reliant on their, rather their messaging services reliant on their production system, which I guess works for most things as long as your networking isn't that thing that fails. Yeah, I wish very well the Google engineers who doubtlessly work furiously over the weekend. Yeah, so, well, we'll leave the rest of that, since there's no changes to the release notes at the moment, so we'll leave the rest of that for next week when we have more of that posted. Code of conduct, so we have, I was looking over at the requirements for things that we're going to have to do in the future, one of them is we need to adopt a code of conduct in order to become graduated, and they explicitly ask us to accept the CNCF code of conduct, but what I would like to do instead of just saying I'm blanket, let's adopt it, I'd like to pitch it this week and next, it'll give time for people to read it and understand it, and then next week we can decide whether this is something, so there's two questions. Number one, is it appropriate? The second question is, is it appropriate to apply it now or should we wait until later? So, if you can open the code of conduct. Sure. Cool, so they... Lots of languages. And so, this is basically a more verbose version of Tess's excellent code of conduct, and so basically is just to, the main things is banning the use of behavior around sexualized language or imagery, personal attacks, trolling, insulting, harassment in both public and private, doxing people who don't wish to be doxed without explicit permission, and a catch-all of unethical or unprofessional conduct. The one thing to call out on the bottom is the section on maintaining a mediator, so if there is a problem between a community member and the maintainers or a problem between maintainer and maintainer, they do offer the services of a mediator, which is fishy chattery. I have not met fishy chattery, so I don't know, I can't say anything for or against in that scenario, but I do have some confidence in the people that the CNCF has brought on for this kind of stuff, so. Yeah, have we checked in with Mishie to make sure, as we adopt this, that it breaks me as a kind thing to let somebody know that they've been signed up as mediator for the project? Yeah, I think we should because, you know, even though it says all CNCF projects and so she probably is part of like a blanket catch-all in that scenario, but I think we should reach out and introduce the community. Yeah, I think that's probably wise. I mean, there's a whole lot of stuff that is just, I mean, we're a super friendly and courteous community and I like that about us, and even though I have no doubt that as a CNCF project, Mishie would stand forward and mediate for us, probably kind just to learn that. Yeah, and as always, if you have any problems that you can also, you can also discuss with us and we'll do our best to help. Yep. And so it's part of the job of the project maintainer is to enforce the code of conduct. It's actually the last line. Project maintainers who don't enforce, maybe permanently moved. So it's actually quite a serious cause. So I want to give people time to read this. Are there any comments as well or anything? Because we're not always, make sure that if there's anything that we don't want applied in here that we call it out as well. I'm surprised it doesn't say violence directly. Well, I have been repeatedly by Mike Dolan, who is an attorney at the Linux Foundation that weapons on stage has been instantly caught in the code of conduct. Or rather, there's a fact that you shouldn't have weapons on stage. So there's a CNCF events code of conduct as well on the bottom, which it also links to. So you should also make sure you read that, which covers, I think, only the events itself. But I think violence definitely falls under, well, I don't know if it falls under harassment or not. I think personal attacks. Personal attacks. Yeah. I think I get the loophole. No, you can have an impersonal attack. Again, I have great confidence that Mike Dolan would consider physical violence to be covered under this code of conduct. But yeah, unfortunately, I don't think there's probably going to be an issue for us. Well, unless the violence was voluntary on both sides. Anyway, cool. So I think it's probably good for people to go take a look and redo this. I've taken some time to look through it in detail and was actually super pleased. But looking through it in detail did lead me to realize the degree to which it's probably productive to give some folks time to look at it in detail. Yeah, so let's discuss this again next week when people have had time to read. If you don't like the way that something reads, I mean, we can do one or two things. We could either choose to, we could either respond back with feedback saying, hey, we'd like to adopt a bit of conduct, but we think this change should be made. We could choose to adopt it anyway. I guess there's four things. We could choose to adopt the modified version. And we could choose to not adopt it and do something ourselves. So is there anything else on this, or should we jump up to the next, jump towards the Andromeda release and start going over the backlog? Yeah, let's move. Cool. Do you have a forward decline? Well, the Andromeda release, as we said, the release branch is already on. You have to figure out the publishing of the images, the proper detect images. Think that we have everything set up there. In terms of, yeah, in terms of the backlog, think that most of the things are already, yeah. And the same with IPv6 payloads should be, okay, not fully result, but we are testing things there. We can just talk about all of them. Okay, yeah. And many, many thanks for the IPv6 stuff. It turns out that IPv6 was, IPv6 turned out to be exactly the way IPv6 always was. It always was.   Yeah, the last one was really interesting. But yeah, we kind of went through it and we have a nice verification in our CI that we can safely pass IPv6 payloads and that's being constantly checked. So that's good. What else we have here? I think that, I don't know, maybe I need to do a pass here again and check what's still on. But from my personal tests and my experience with the examples, they're, I mean, LSN behaves pretty much stable for the type of project that we, I mean, like the status of the project that we are. I mean, I don't see any obvious, huge crushes and misbehavior. I think that this is, of course, due to the extensive integration testing that we're doing and all the work that we did in the last couple of months for enabling other clouds which are introducing other problems in terms of delays being slow, being fast, whatever. But then in the end, we obviously have tested a lot. So at least from my point of view, whatever we have in the release branch is something that we should be proud of. So thank you all for putting your efforts with patches and with whatever you were helping. I mean, particularly to all the folks, I know that it's been sort of the steady march of little tiny issues. And then some, as we said, some bugs in the CI environment. Turns out the code is fine, but you know, the AWS cluster never comes up. And one other quick question though, because I know we've met you on the phone, Matthew, you have a real Mac for running into, you test things differently than everyone else. How is it feeling to you? I think it's fine for now for the 0.1. It runs pretty well. And no one is expecting it to be run for production. So it's fine for me. Yeah, when people ask me about that, I usually put it obliquely and say, we're about to do our 0.1 release, which is exactly the way you would expect a 0.1 release to be. Yeah. Awesome. The good thing is that people are starting to appear within the issues with requirements or some observations. And Matthew, I guess that you have seen, there was a post in the group about skydive, some, so yeah, I mean, obviously people are testing us and I think that we are more or less ready for this. No, I'm feeling pretty good about it as a 0.1 release. And we got lots of exciting stuff to do as we move forward past 0.1. And I'm super happy that master is open again. That's always exciting. Yeah. We're going to break it soon. Oh yeah, sure. I mean, hopefully not break the existing stuff, but yeah. Yeah. I mean, adding more tests and stuff and yeah. So, I think that we should move. Do we have Ramki on Prem on the call? I mean, that's for me, for the Andromeda release. I mean, I expect that before KubeCon China, we should have attack and published images, something that can be shown around. That's my expectation and plan. Cool. Awesome. Do we want to talk road back going forward? Cause that's always super fun. Yeah. I don't see Prem or Ramki on. I can ping them and see if they can, I can ping Prem and see if he's willing to hop on while we talk about other things. So the roadmap, yeah. I think that we said this last time, I would really like to keep this item here for us to remind us where we are, what we do, maybe add things, remove things, kind of checking quickly the status. Just kind of open discussion here. So release is clear more or less. The examples we set, it's kind of a little bit tight with the next topic, CNCF test bed, and also me and Taylor, we were just having a good, nice conversation, actually preparatory conversation for the calls to have this week. So we have scheduled a number of calls with people that are interested in enabling CNSM for the CNCF test bed. We have some rough plan there. I would really like to see this enablement based on the examples, and I don't know, we'll have a discussion tomorrow. We'll figure it out if we are going to push the code there, are we going to push the code in CNCF? Okay, we'll see this. But the idea is that the examples for me are becoming the, should be from my point of view, the starting place for everyone that wants to use, and so wants to deploy. I also think you were talking at some point about, we've got a bunch of the examples living both in the repo, in the main repo, and in the examples repo. And I'm also talking about possibly moving the examples of the examples repo. Because most of the time those examples don't change. And so you should just be able to go around. Yes, that's the point, yeah, yeah. But that's a conversation that we probably should take offline. But I mean, I can't have it now because we have today merged the PR where we are renaming there's no examples folder in the main repo anymore, right? I mean, we moved everything to testing, the sidecar container belongs in the sidecar folder. So everything is moved to other places. So examples are existing by the virtue of just being part of the history and also being part of our CI. Maybe we need to figure out some plan for completely removing the things there. But okay, this is something that we can discuss. It's like also, mm-hmm. Okay, cool, cool. So for Mechery, I know that Lumina were supposed to have something there. I don't know what the status there unless Prem peers or at least give us some name that can join, yeah? Yeah, Prem sent me a message that he's stuck in a meeting at the moment. So we should probably punt that out next week. For SMI, we are going to keep an eye on it. I mean, on our side, on my side, but for time being no active movement there, it's a bit of a convoluted topic. I know that people have opinions. Yesterday, on the technical, not technical, telco user group meeting, and actually today, someone, I don't remember name, the name, but there was, I mean, there was an initiation of a white paper like preparation document. So SMI was mentioned there. I don't know how this merges, but I think that we should just keep an eye. That's enough for now. Colonel Forwarding playing, Raduslav, who is here, he started doing something. As I said in the chat today, I hope that we learned some things along the way of enabling VPP with IPv6 and debugging things there. So I think that we learned some things. Maybe we have still a lot to learn. I don't know, you'll see, but we're definitely on the path to move this forward. I hope to get some patches for having, like today, VPP is very deeply embedded in our testing, integration testing. So we're planning to have some abstraction there so that you can switch back and forth between the different implementations, which I will hope will make some people feel a little bit better because I have heard some opinions about this. Everybody likes having flexibility on the data plane. Yeah. And I would really like to stop calling it data plane because that's not what our glossary is saying. Okay, reporting play. Yeah, maybe we need to rename things also, but... I'm always open for re-education. Yeah, we can use the term subnet provider. No, let's not use that. Let's not do that. So security has been a thing that we've talked about a lot. And I don't know.  I've got some advices ideas, but I don't know. I mean, I'm ready to participate in discussions about it if this exists, but yeah. I scribbled something down for security that you may want to go take a look at, which sort of looks at it and says, okay, let's sort of look at how we handle identity, authentication, proper authorization. Not only for like the current single domain case that we have, but looking forward to a multi-domain case as well. So don't have a look at that. See if it makes sense to you. You know, and see kind of what makes sense. Part of it came out of comments from conversation with QCon where folks were like, look, the Spire Guy, the Spiffy and Spire Guy is just about Sol Federation, which is gonna be super important. So I'm happy to hear or someplace else talk through a little bit of that, my thoughts there. And I'm very much open to other thoughts and ideas. And please God review because security is the kind of thing that the more eyeballs you get on what you're doing, the better off you are. So this might be something worth eventually setting up some form of a technical work group specifically to go over security things on a... So I know that there's security and actually I popped in one of their group meetings there at QCon and they were saying something that they are doing things like project evaluation or security evaluation of the project, et cetera, et cetera. Like audits, but they can also do kind of pre-audits just because there appear to be people that actually are deeply into security so they know the common mistakes that people make, et cetera, et cetera. I don't know if it's worth involving them at this stage but I'm just saying that we might want to think about it at some point. Yeah, I mean, so I actually bumped into Justin Kappos while waiting for my flight and he's involved with the CNC up doing their security audits. And so what I probably, you know, minimally I would appoint him to the current scribbles. So that he can take a look and see what his thoughts are. And I've talked to some of my internal security folks about the general gist of it. I even done the security fellow here at Cisco. Yeah, and he was positive-ish about the issue. Now admittedly he then went on to give a talk about how you continue to secure things in the post-quantum computational world, which was interesting. Okay. So, I mean, no one vantage point necessarily here you've gotten it right, but the more eyeballs we get the better. Which actually reminds me that if we have these scribbles maybe I can post them to someone on our side too. So that's good. Yeah. No, I mean, we, we all, you know, for the most part we all come from places that have some version of really deep security humans around. True, true, true. And so you agree that we can it, it moves us to get them to weigh in. But I was particularly heartened by the security fellow being positive-ish, especially since he has a history of... So it means that we are at least obviously moronic in the direction we're taking. Okay. And then I think you, you'd written up something on DNS, Frederick. I have. And that sites that Google Doc, you open it up, shows the initial... So what I would like to do is I'd like to, let me give some background. So one of the things that we were looking at was what happens when you hook up, let's say multiple VPNs, or even just a single VPN. So now you have two or more DNS servers that you have to resolve against. And so one of them being your primary Kubernetes resolver, plus any additional secondary DNS resolvers that you attach on. And so this one of the solutions that we're looking at is if you look at the second image, so is adding in a core DNS sidecar. Well, we say sidecar, that doesn't necessarily mean that it's a container in the same pod. It could be a, it could be another pod that is running a core DNS in that area. Actually is that true? No, it has to be in the same container because of the game space. But effectively we want to set up a core DNS. We're looking at setting up a core DNS sidecar that is configured to use all of the VPN servers that you care about. So first, I'd like to talk about whether even, whether this is like a sane idea or whether, or what problems we can run into before we jump into the provisioning side. So does this look reasonable to people? Does this look like something like where we're working to sing break? So the one that I think I've mentioned before when we discussed this, that's a subtlety that's quite fixable is, I mean, obviously if you've got a core DNS sidecar that's fanning out the DNS requests to not only the normal Kubernetes DNS, but possible DNS is upstream of the various network services. Normally if you get a positive response, you would actually like to pass through the first positive response you get. You do have to be a little bit more cautious with negative DNS responses because you don't want to pass through the first negative DNS response you get because it may just be that somebody else has a response that's positive and they haven't gotten it to you yet. So when you say first response as well, is that literally like the first response regardless as to which one it is? Or do you mean first as in like the Kubernetes would get priority and that's considered to be the first response and then VPN one, regardless of the time ordering would be like the second response? I was thinking in terms of time ordering because you get very long latencies if you run through them serially, it's much more efficient to run them in parallel. And the Kubernetes DNS is going to have a huge latency advantage in that not only is it running in the cluster, but generally speaking, the Kubernetes DNS is now moving to per node caches for DNS. And as a result, you would expect that if Kubernetes DNS has something to say about it, it will have something to say about it really fast. Okay. So something we should probably consider in this scenario as well is making sure that we, for certain domains, if we can say like we want everything from example.com, like all requests for example.com to go only through VPN one or VPN two. So I think we should, I think it would make sense to add something in like that just so that you can say things in my private corporate internet. You're not pulling the public addresses, you're only pulling the private addresses for those. Probably supervised, especially considering people who run all kinds of crazy split DNS schemes. I see them way too often. Yeah.  But yeah, everybody does it. So, but I mean, what I would encourage, because we're getting down to time here shortly is folks to please take a look at this and comment on it. This is going to be super important for us to support. Going forward. Yeah, I've added additional provisioning steps in there as well. So please go over the provisioning steps and, and let us know where it works and where it breaks. Yep. And I'm just good for consistency. Just make DNS a link there. Cool. There we go. Cool. So the inter-domain stuff, I'll go back and for next week, I'll put a link to that. We've got a spec for resiliency. Two is right now we've got a good story where if anyone network service manager or network service manager data plane dies, everything reconverges. And if you lose a network service endpoint, we auto heal. Andre had some really smart ideas where we could literally lose all of the network service managers and all the network service manager data planes at the same time and still reconverge, which I think we're tentatively calling resilient CV to folks have other ideas for naming. That's awesome. Dynamic rewiring. So we have auto healing already in the system, meaning that if you lose a network service endpoint, we can automatically reconnect a client to another network service endpoint to continue good continuity of service. And we know we can use the same mechanism so that if you change the networks, the policy for the network service, we are capable of dynamically rewiring that mechanically. And I think there are even ways for us to do it safely because obviously you don't want to do it to everything all the time. That could be destructive. And that might, that should allow us to do a really cool example where imagine that you have a pod and it's consuming a network service and something is weird and you decide, you know, after the pod is running for a while, you wouldn't it be nice if you had a packet capture box between the pod and everything else that was going on. Potentially you could use dynamic rewiring to do that. And you could use something like web shark to see what's really happening on the wire. And I don't know the developer who doesn't want that capability. So specifically what I'm hearing is you can use a web server to display your captured packets that you injected with the listener you injected directly into the network service wiring itself. Yep. Yep. Cool. Yep. So, and then the hardware and accessories. Two things I do want to make sure is number one, I've been starting to scribble and together an attempt at a network service mesh technology tree because some of these things do depend on each other. You know, so for example, security has to be before we can do inner domain. And I've been trying to collect the various things here together so that we can at least sort of try and put some order on all of it. And in doing that, one thing that became immediately obvious is we have been collecting specs on the spec board. And it might be a good idea for us maybe today, maybe next week to go through the spec board and make sure that we capture those things and that we're getting them all organized probably into the technology tree and other places. So we at least know what the bogies are. Right. What are the things we could go work on that people might want to go work on in the community?  And I know we got some of that done, but I don't know if we've got all of that done and like Qvert integration. I know you were interested in that Jeffrey. I'd want to make sure that we got that represented correctly. We've got some new interesting things coming in around network or load balancing where people have got suggestions for that. And so I just wanted to make sure we sort of walk through the spec board, make sure we were capturing all the specs and getting them filed correctly so they could be worked on if people were just working on them. Make sure you ping Jeffrey remotely because he had to dash. Okay. Cool. Awesome. So. Yeah, so that was that was what I did there. It sounds like Nikolai, you thought the technology tree was potentially useful. Yeah. It was awesome. Also, also you're using the polite like the logo polite. I guess, like the colors. Yep. Yeah. I'm using the new colors and actually I'm, I'm not sure why, why the capture NSE needs dynamic rewiring. I mean, I can see but I mean, I can see the use case where you kind of can inject dynamically the packet capture, but the packet capture NSE can be part of the already deployed service. So if you want to demo that. There's a, there's a cost to injecting a packet capture in. So it's not a free thing. So you don't need it in order to, in order to do the basic use case, but it becomes a lot more useful if you can rewire it in. I think part of it, because I know you've been working on this Nikolai in the sense of building and our service at point, and I think we're suggesting is it would be, you could just build that in the network, the NSE in with supporting a packet capture network service. That's something like the web, the web chart could connect to, and that's actually probably true now that you mentioned it. I see. So having the NSE itself be able to inject it in at the appropriate moment. Yeah. Or just have the NSE provide the network service for it. Yeah. Fantastic idea. You know, and so you can sort of sort that out there. Of course, obviously some interesting things about there. But, you know, I think they're relatively solvable within the framework that's emerging. Yeah. That, that, that means that may actually be true. We may not actually need this connection here. Okay. I would have to jump to another call. I'm sorry to the, I'm a bit busy. So I would have to leave you guys. And since we don't have Jeffrey to talk about data plane separation this time around. Yeah. Unfortunately tomorrow, we have an overlap with the CNCS. That call for the doc call. So I won't be able to join the doc call. I hope that someone else will be able to. I will cover it. Yeah, for the, I did have a brief talk with Jeffrey about the split data plane. And I think the concept was that you may have two data planes that have different capabilities that. Would be complimentary on the same node. And so the question would be how to make sure like, if one of them supported. My life and the other one supported. Sorry. Then like, how would you, like, how would you forward them on to the right things? So I think it's based around that just as a, as a teaser. But I'll. Let Jeffrey talk about it next week because he has a more concrete use case in hand. Yeah. And quite honestly, there's going to be a little bit of that going on as well when we get to the hardware stuff. Because one of the things that's going to be super important is to be able to enable people to be able to provide the thing that handles the programming. So that we don't have to, and they don't have to. Okay. Yeah. All right. Anything else guys before we call it a meeting. Just, just about the roadmap. It's, it is a roadmap for the zero. Two version.  Priority. Yeah. So I, I think it's just the roadmap in general. We just got zero. One branch. And so I'm not sure. Yeah. Part of what lands in what release is going to depend on who's interested in working on what. And when. But the thing is, if we have a general notion of a roadmap, one of the things we can look at in zero. Two is do we want to do a time-based or a feature-based release? And if we do a time-based release, then we'll sort of say, okay, we're going to release around here and then we'll see what we can do. Get in by then. And, you know, the things that will drive that will be the priority with which people are interested in working on things. Plus. By the technology tree. Like what has to happen before what else? Cause. Interdomain without security is just not going to. And then. We get sort of. There. I think. Which regardless as to which direction we take, we should probably do releases relatively often. Just so that we can get into the habit of doing the release and getting good at them. Like if it's something that's going to be different, it's going to be different. So. Yeah. What you, what you do routinely works. What you do occasionally often breaks. So if we only do one release of end of summer year, we're going to go through a lot of pain in getting it out. But if we do lots of little releases, then the little releases will be. Mostly painless over time. Yep. Yep. But that's, that's probably a good conversation topic for you. Yeah. So. Thank you guys. It's been great. Thank you very much. And with that, close it up and see you all next week. Same time. Thank you. Bye. Bye everyone. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye.


Hello everyone and welcome back to enter the ether or the podcast all about the very ambitious upcoming third person mobile ethereal clash of souls. We have a lot to talk about this week. Fortunately, I have the best in the business when it comes to talking about this stuff. It's my friend and cohost jelly knees. How you doing jelly? I'm good man. I'm happy to be here and ambitious is like an understatement with all the things that we've learned recently.  They did say that they're willing to back off on some of these things if they don't quite work out, which I think is important for them. They need to they need to back off on some of this. They're going a little too wild. I think the main thing that I feel like they can't back out of at this point is the classes, right? Everything else that we've talked about and what we will talk about today, I'd be like, OK, if it doesn't work, it was a cool attempt, but it's OK, right? The classes are the one thing that I'm like, please, this has to work. This has to be the thing. The classes in the in the map, I think that's I think that's the main draw for ethereal is the vertically tiered map. Their their lanes are stacked one atop each other. I don't know if I can really indicate that with my hands. OK, so as jelly figured out and theorized, they did have phase two for their website, which at first. Not that impressive. Like when it first came out, jelly and I were talking about like, what is this? Like, it was kind of our fault because we'd hyped it up so much in our heads. And they never said that it was going to be like big blowout thing. But when it came out, a lot of us were a bit let down. However, as they've added more to this phase to like, this is an ongoing phase. It's gotten to be a far more impressive. And they've released a lot of new information about the game. So we're going to dig into it this week. Jelly, before we do that, before we do that, the podcast is now officially a podcast instead of a vodcast. Oh, yes. And we're on currently Spotify and Stitcher are all the episodes. And then we are applied for Apple, Google, everything else under the sun that we're just waiting to hear back before we get on to those podcast areas as well. So if you want to listen to it, if you're in the car or something like that, you're going to be able to do so from now on out.  You can follow your laundry and listen to the latest ethereal news. So in we'll have links to what we have currently available. So Stitcher and Spotify in the description this week. So if you then you can get that going forward, follow it, do all the listens, downloads, clicks, I don't know what podcast terms are. So those things. And yeah, so let's now now this week we can actually do the official thing. Indeed. All right. So the phase two at first, what it was, it was just kind of additions to they have their in their discord, they have the development fun facts. And it was stuff that was just kind of adding onto the development fun facts. But as we've gone along, there's been a lot more information come out in this phase two of the website. So if you go to the undyinggames.com website and then you click on the arrow at the top right hand corner, it'll take you to phase two. And there's these blog posts that are very reminiscent of what you would find in like a Kickstarter campaign. If you go through and look at like the various things that they're they're saying that they're going to bring. And so let's let's start with the first one, which was November 13th, two thousand twenty, which addresses addresses, special abilities, something we've already talked about, which is the right click abilities for each hero. We'll have a how would you call it, Jelly? Pros and cons, I guess, is probably the easiest way to put it is a benefit and downside.  The big thing that they did on the website for this post is they gave some like hypotheticals of what abilities like this could be for the both the pro side and the cons.   They gave five examples of pros and five examples of cons. So in my mind, these are actual abilities of some kind that we might be able to tie two myths that are that will be in the pre alpha or come out in the future. Yeah. And some of the benefits include movement speed, more damage, additional gold decrease, cool down on abilities and more. What? Don't touch that dial. Yeah. And then the cons, whereas it cons will decrease your movement speed, blind yourself in place, allow increased damage from other sources or apply extra resources to cast your abilities. So to me, it seems like those all tie to each other one to one. Yeah, that really does seem that way. One thing that I thought of immediately when I read this post when it said blind yourself in place was Richter, as you guys all know from Paragon, I was a huge Richter fan, love playing Richter. And that was one of the drawbacks of using his ultimate ability is you would just root yourself in place while it was going off and people could just tee off on you. That's a pretty big drawback. So whatever they make, whatever the the benefit, the upside to using that ability is better be pretty strong. I completely agree. And it's so for that one in particular, I feel like we're I'm getting Noxus vibes that that's going to be Noxus's right click. I don't know someone. What makes you say that? It's the blind yourself in place because she's blind. Not not necessarily that, but yes, at the same time, because it's when you're when you're blinded, you can't see, you can still move. Whereas she's already blind and she uses the light around her to be able to see and move. So if she doesn't, if she loses access to that for some reason, then she wouldn't know how to move kind of deal. I don't know. Could be a complete long shot on that one, but. That's you jelly all over with the tin foil hat conspiracies. They need to make a tin foil Noxus where she's got a tin foil hat covering her eyes. That's what they need. Perfect. The one I think is really interesting is the do more damage and then allow increased damage from other sources. Yeah, it reminds me of what was that card in Paragon where it would increase your damage, but it also increased damage taken and would do it in like an AoE around you. Yeah, I don't remember the name of the card specifically, but I know I remember its existence. And that's what it sounds like. That's strong. And I wonder if that's more of a consistent effect, like it's a toggleable passive that I turn it on, I do more damage and take more damage and then can just turn it off whenever. Or if that's a actual ability that only lasts a certain amount of time would be my curiosity, because if it's a toggle, the best player for whoever that is is going to be the person that can toggle it on and off in perfect sync with their abilities to get the most effectiveness out of it while taking the least damage during the process. Yeah, if you got stunned while that was toggled on. I screwed you be screwed.  Sorry, like. Bye-bye. Just the stun itself would hurt. And additional gold is an interesting. Advantage or disadvantage. We've seen that in Paragon with Revenant. Yeah, that if you kill and I'm curious if this applies to all income sources or just myths or just minions or what, but just additional gold when you use the ability or all the time, like it's just. Yeah, I don't know. It's this plot implies a whole lot of interesting mechanics that they might be able to do or even other it opens the door like we were saying before when we were talking about the right mouse button abilities is that once we have an idea of where they're going with the pros and cons, it'll open up a lot more of what we can expect from those abilities. And this spectrum kind of covers it all. That like there's no ground that they're not willing to go to for one of these right clicks pros and pros and or cons. Right. I didn't even think of additional gold from like minions or something. I just my mind immediately went to Revenant when I saw that and that's what I thought of. But yeah, you're right. It could be just gold from all sources that you get an extra gold per minion you kill or something or when you kill with this ability on minions, you get 10 gold, whatever it may be and a little bit of gold goes a long way. So it's it's going to be very interesting to see how that ties into everything else. Yeah. You ready to move on to the next post? Absolutely. Next post was about skins and it was kind of just talking about a lot of the stuff that we already knew about skins that even their tier one skins will be transformative to a degree. However, they did include a picture which had quite a few skins that we have not yet seen. And a lot of them are really cool. Jelly, what do you think? I think these are incredible. The one I have the most concern about is the Dante skin, but I feel like that's because all we can see is his hood. That's the like I want to see the rest of it like what's what's missing that I can't see in the rest of the art. Yeah. It just looks like his base skin doesn't it? Yeah, a little bit. But the other six incredible.  And that that skin looks amazing. But even Talos has got like hardcore monster hunter vibes going with the extra horns everywhere in the fur and everything. Yeah. Yeah. Cyberpunk Malaya we already knew was a thing, but it just looks amazing. I like I'll never get over how good that skin looks. Marina skin changes up a lot of her outfit and the coloring of it and it looks really good. Like I just think it ties together really well.  Like it's just I'm ready. I'm running at me in a lane. I'm stoked. You mean Iran like his curled horns and stuff. Yeah. Sorry. Not Talos again. And then Noxus, I got my dream. I got the dark skinned Noxus. Yeah. It just looks so clean. I just love it. So I called it on Egyptian Leah and you called it on a dark Noxus, which is pretty cool. I think the one that I'm impressed most by is Marina. Not because of any normal mangoes reasons. I just think it looks really good. I like the red hair on her. I like it's kind of a succubus sort of feel to it. I really like that. I really like the that skin for Marina. What's your favorite skin about out of all of these here? I have to go with Egyptian Leah. Yeah. Yeah, I just I look at that. I'm just like that. I'll buy that second one second in. I'll be like, yep, that's the skin I want. Like I don't know what it is about a base skin, right? It does. It absolutely does. And it gives me spite vibes a little bit just because that's because the Egyptian theme. But I just love the way it looks her there. Her whole style there. It just looks incredible. It's something else to talk about with this. So as you see in the picture that I have a bit right now, you see that there's these strange runes at the bottom. We have deciphered a lot of those runes between jelly and I. And you've been working with somebody in the discord. Who is it? Who was that jelly? I can't. That was Young Boomer. I think he changed his name to Torquey recently. So I don't know what he's going by right now, but that's who was working with me on it. Yeah. So we've been deciphering these runes. We figured out quite a bit. So like we can tell you right now that those runes right now say cosmetic. And that's kind of what what the starting point was. I haven't contributed much to this. All I contributed was I figured out that this was cosmetic. And the jelly and Young Boomer went from there and like the cyber and everything. We went hard. I mean, the only letters we're missing right now are B as in Boyd, D is in David, F is in Frank, J is in Jelly, K is in King. I almost said Castle. V is in Victor, W as in Winnebago, X is in X-ray, and Z, Zebra for all you or Zed. Sorry. I should say Zed for all the non Americans in the world. Those are the only letters we're missing. We have everything else figured out based on this image and then a couple other images. And Young Boomer and I went back to the extra life play through and around the towers. There is the same like lettering around it and we went through and grabbed screenshots of those and deciphered the wording around that, which let me pull it up really quickly. The towers around the towers, it says nothing real can be threatened. Nothing unreal exists. And we believe that that loops twice. Nice. That is amazing that you guys went that far in depth and figured all that out. It took a while to find like a good shot of the towers that we could actually decipher what the letters were going around. Yeah, I was going to try and help you guys with that, but I couldn't find a good shot of the towers. I was like, I figured the cosmetic thing out. I'm done. You guys take it from there. If I know jelly knees, he'll figure it out. It's fine. Exactly. Exactly. So let's move on to the next post, which is the minions.  These blog posts on the phase two of their website tell us a lot more. So jelly, why don't you tell the folks about the minions? So I was completely wrong. I'll admit it. Uh, these, the minions will actually have classes because I didn't think that was the case at all. Um, the minions will have each, there'll be eight minions, right? I'm not crazy. One for each class and the plan is that when it all comes to fruition, the minions will all have a special ability based on their class of some kind. They didn't go into any detail about what that ability would be. Um, but the standard minion wave will be a night minion, a cleric minion, a reaper, an archmage and two marksman minions. So three melee, three ranged and that'll be your wave of minions. Every three waves, I believe is what they said will have a Berserker minion that comes through kind of like your siege minion. And then when you destroy the inhibitor or Phoenix, depending on what they're going to call it, you will spawn sky slayer minions into your lane, which sounds incredible and terrifying at the same time. What was your thought on the minions? The way, the way it was stated, the sky slayer minions, it won't actually be sky slayer minions. It'll be the regular minions, but they'll have a sky slayer buff. So they'll hover off the ground a little bit, which will give them increased movement speed and damage, I think. Uh, yeah, it does the increased damage. Yeah. So it's not actually a sky slayer minion. It's just the regular minions that are already there. They just happen to be hovering in the air, which will make them a lot stronger. So really cool stuff. And this, um, I must say that this is where we were talking about before. They did say that if this doesn't work, if this is way too hard to balance, they're willing to just drop it. And I'm really happy to hear that. I would hate them to try and force this to the degree that it's just broken. And I can see this getting out of hand real frickin fast. Absolutely. Like it would just be who could take the tower the fastest to get those sky slayer minions and then just jam those up a lane. I can see that happening so fast. Absolutely. And I think this, at least on my end shows a different side to a theory that we didn't see towards the end of Paragon's life. That if something wasn't working, they're not going to just keep shoving it down our throats. They're willing to make changes and make the game better because of it. Like it's a cool thing, but if it doesn't work, it's not the basis that the game is going to be built on and what they want the game to be built on. But I agree with you. The balancing of now just minions by themselves, each having their own ability, each getting a sky slayer buff, the, like the different classes is going to be a nightmare, I'd imagine. I don't envy anybody that's balancing those minions because that would be insane. And just creating all those extra animations and every an extra models and everything has got to be very taxing on the team. I can just imagine that we'll get into game initially and Berserker minion spawn and they come up in just one shot somebody. It's like, nope, nope. Too strong. Way back. I do like that. So a Berserker minion, I guess, would spawn towards the front of the wave as opposed to the back of the wave, which is what you normally see from a siege type minion. That would make sense because it's not ranged, I wouldn't imagine. Right. But I know we could have ranged Berserkers, I guess it depends. Part of this is it all depends on their abilities. Yeah. You would assume based on the fact that they're called the same as the class that the ability would tie somehow into the same thing as the class ability. But who knows, it could be anything. I was really curious if they do, if they make the cleric minion have some kind of heal for the wave. That could be a really interesting thing that your minions have a healer that you can destroy the healer and now it's going to be easier to push that wave in in comparison to trying to keep them up and pick them off last. You could set a slow push by simply killing the cleric minion and that's it and then leave that lane. Yeah. That's very interesting. I never thought of that.  Just build a giant wave and crash down town. Yeah. Absolutely. I feel like the wave with this wave management is going to be a huge thing that if you know how to manage your waves well, then you could dominate games with just that. And wave management is a very important part of Mopas that I think is missing from a lot of the Paragon remakes right now. And I really want to see it come back. And with this, I think it really can. But I'm also happy that they're willing to just drop this if it doesn't work because it can pick and very well not work. Absolutely. I like to see that flexibility. And then the image on this one that hopefully Mangus has been so kind to put above us has that same writing underneath that says protect in their own language, whatever you want to call it. I kind of want to call it just a therian in a therian. It says protect, which I mean for the minions is I think a good word to put there. And it gave us puzzle pieces. Yeah, which is nice. It was really that image that kind of solidified like figuring out almost the entire alphabet. Once we had enough letters, we could piece words together being like, OK, only two letters are missing here. The word must be nothing. Like here we go. And it was so much fun, but it was also so ridiculous at the same time. When they were releasing all this, I mean, I was in the discord like every 10 minutes, found a new word. Like here we go. And I like in the discord, they were like, oh, you guys think you figured it out. I'm like, yeah, that's what they're saying like in the discord, but like in reality, they're like, holy shit, these guys figured it out this fast. They're like, OK, jelly, someone needs to put a leash on jelly. This is not. Nope. You can't do this. They finally acknowledged it. They're like, oh, OK. You guys got it. OK. And so the next thing for the blog post that is just as big as the minions, in my opinion, is the item system, which this is really the first time, other than hearing how many items and that there'll be actives and passives. This is really the first time we've gotten actual information about ethereals item system. And so this is the one for me that I was most excited to read because I was like, OK, this is this is the basis of your game more than anything else. How is this going to work? And they I mean, they've done an incredible job and I'll let I'll let you explain it, man. Goose. OK, so if you look at this image, you see a one, two, three, four. Those are obviously active slots because they have potions tied to them. However, you also see seven other slots. So those are seven slots for equipment and you have four slots just for active. So you don't have to take up an equipment slot with your potions or your wards and all that sort of thing. And then the one thing that they mentioned in here is boots.  If everybody's already going to take those boots, the boots anyway, they've solved this problem by having the boots be a seventh slot that is free to use and only four boots, it seems. And so, yeah, so not so it is kind of standard all across the board, like the movement speed, but you still like a lot of the argument against that is that a lot of times you pick different boots according to your class that are better for you. So you still get to do that, but everybody still gets the movement speed so they can balance the map around movement speed from boots while still having extra stuff from the boots. Amazing. This is an absolutely great solution to the problem. I messaged Owen. I was like, hey, man, this is a really creative and great solution to the boot problem. And that's not it. Jelly, there's even more. There's even more of this item system. There's different types of items. You want to speak more to that? Yeah. So the system itself is going to be a tiered item system that the tier ones will be your cheapest, weakest items, and then your tier threes will be your biggest and best. And your tier ones will build into tier twos and then build those will build into tier threes. So kind of your standard build layout in the MOBA genre. But the crazy thing, they said they're going to have gauntlets, gauntlets, charms, guns, marks, masks, rings, and sides as a few of the item trees featured in the game. So everything in that tree will be whatever that tree is about. If it's a mask, all tiers will have mask items, which is an interesting progression because you see in other MOBAs, for instance League, just being the one I play the most, you're building the attack speed sword, but you put two bow items into it. And so it's this weird, like if you think about it, you're like, how do two bows equal a sword? I just, I'm not figuring this out here. Whereas this combats that problem entirely and just is like, nope, they're all swords. Like just across the board, your attack speed sword is just a sword all the way through that just has other stats, passives, actives, and a look to it, which incredible. I think this is a great design. I echo your point about the boots. I think that's a whole new level of innovation for the genre that you still get the stat changes, you still get the passives that probably apply, but you also get the movement speed without having to make it standard to just every character across the board. And it's something they can tweak accordingly. And I just think all around such a great choice for this game and in general. There's also going to be jungler and support specific items that anyone can build. You don't have to be in that role to build them, but that's kind of the intended role for these items. So the marks and masks are created in order to be items for junglers and supports in their role. And I think that that's a great thing. I imagine that the jungler items will be something like an execute or a smite for the jungle minions, potentially buffs of some kind or a way to get buffs from these minions. Support items I would imagine are the boards or some kind of vision access and extra gold for supports since they're not going to be getting the items or the main gold from minions from their carry lane. But honestly, they could be anything. They've said that they're not going to restrict them to the jungler role or the support role. So it could be anything, honestly. I'm stoked as a jungle main myself. I love this idea. I think it's great. And I just can't wait to see it. And as a support main, I concur. This is one of the really good things we've seen come come out of a lot of the Paragon remakes is they've really come up with creative ways to help supports and junglers. And apparently Undyne Games has learned from this and they're applying that to their own system, which I'm really happy to see supports just get shit on as far as economy. Like you can never build your full build as a support because you just aren't gaining as much gold as the other classes. And then junglers, the smite, they always have to waste something, waste an item slot on a smite to help them clear the jungle. So really great ideas all around. I really like their ideas for the item system. Absolutely. And I can't wait to see the whole, they mentioned in the discord that the unique passives and actives for these items. So I should say you can't build multiple of the same item or you can build multiple of the same item. You can't stack their passives if they're unique. So that's pretty standard across the MOBA genre. But they also said that the unique passives and actives are going to be so strong, useful, whatever you want to put here that you won't need to stack multiple, even if you wanted to. Like it's going to be that interesting and unique and dynamic that I can't wait for that because that's it should feel meaningful to make my item purchase more than just a stat change. Yeah. If I'm purchasing an item for the passive, I want that passive to be I'm like, oh, that passive definitely did something there rather than just like I did 32 extra damage. Yay. And it creates interesting builds that are personalized to how you want to play too. Absolutely. Really good stuff all around. All right. I think that covers everything. Do you have anything more to say on the Phase 2 or or on the language that they speak or anything like that? This one, the item image says equipment and was kind of the definite final puzzle piece to solidifying all the things that we thought we knew prior to this. But incredible. These updates were really nice to see more in-depth versions of everything they do. So they will update the fun facts on the Discord Friday. It seems like Friday mornings and then they update the blog post with the more in-depth version on Friday afternoon. So they kind of give you half a day of like, what does this mean? What could this possibly do? And then answer your questions later. Maybe not fully answer them, but they give a lot more information at least. Yeah. Absolutely. All right. I think that's going to do it for this week. Jelly, you got anything to plug? Are you going to do a video on any of this stuff? I was kind of thinking about doing one myself. I will probably do an individual video on all of these things. I was debating that today, actually. So I'll have those out over the next couple of days kind of going as in-depth as possible about each of these things. And of course, putting on my tin foil hat for some theorizing as the jelly knees does. But that's about it. Podcast actual podcast locations Spotify, Stitcher at the moment. More to come. Yeah. Please take a look and subscribe or whatever you do with podcasts. We barely know how the YouTube part works, let alone trying to be like, podcast download, listen whatever. And then yeah, I might do a video about this stuff. I'm still not sure. I've been pretty busy. But I do want, if you guys have any interest in Kings Hunt, I did a excuse me. I did a complete breakdown of Kings Hunt recently. If you guys want to check that video out. But yeah, I think that about wraps it up. Unless you got anything else, jelly. Nothing comes to mind. All right. I hope you guys enjoyed today's episode. There was a lot of information that we covered. A lot of very interesting things coming out of ethereal. The hype train is still rolling along and we hope that all of you join us as we finally enter the ether for now.  Have a good one. The best outro yet. Right. Mangoo's


So I work on the storage side of things on Taikevi. And prior to this, I used to work for Oracle on my SQL. And I was the lead on InnoDB. And recently, I joined ThinkApp. And my experience is limited, but I do work on the storage side of Taikevi. And this is my colleague, Andy. He's been working longer on Taikevi. And so he'll take it from here. And we'll talk about, you know, how we make Taikevi really more cost-effective. Hello, everyone. My name is Andy. I'm a software engineer of ThinkApp. And I work on Taikevi. Prior to ThinkApp, I worked for Google's banner team. And so here I am. Our topic today is about making Taikevi our distributed storage engine that is cost-effective on the cloud. So here's the agenda. At first, I'm going to give you a brief overview of Taikevi. And then I will show you some of our own experience building a cloud service. And yeah, that's it. And then last, I will give you some examples like how we do cloud cost-effective works. And as well as some works we're doing and we'll be down in the future. So first, what is Taikevi? So Taikevi is an open-source distributed and transactional K-value store. So it's written in Rust programming language from day one around like seven years ago. So we were so glad we made the right decision and it was a tough decision to make. And since it's Cubicon and CNCF conference, I'm honored to mention that ThinkApp donated Taikevi to CNSF Foundation like three years ago and now is a graduated product of CNSF. And I'll give the link below. That's our GitHub link. You can search it online. So, you know, Taikevi as a storage engine has been globally adopted by many users in many different scenarios. So here are some examples. You know, we are infrastructure for infrastructure. In some cases, like there's a file system called JuiceFS that use Taikevi as their metadata storage. And the GED cloud object storage also use us as their metadata storage. So there's also like a blockchain company called Harmony. They use Taikevi to store the blockchain data as well. I guess it's probably the super node or something. I'm not sure like whether you guys play mobile games or not, but I guess, you know, like Pokemon Go and the company behind Pokemon Go, Niantic, they use Taikevi as a transactional key value store. So, you know, we are open source project. So we have a large community and like many community members did a lot of great things like around Taikevi. So they build a radish product on top of Taikevi and they're called like Tidies and Titan. And of course, like Pincap as a company, we provide like a solution, like a query SQL, MySQL compatible database that is called Tidp. That's our main product. And it uses Taikevi as an underlying storage engine. So, yeah, this is like an overview of Taikevi architecture. So this in Taikevi is partitioned into multiple charts and each chart is like replicated across a multiple nodes and a rough algorithm is used to maintain the consistency between replicas.  I know it's a terrible name. Some people like misunderstanding this with cloud region, but this is a rough group. And in addition to raw QV APIs, we also provide transactional APIs. And the transaction implementation we use is called Percolator and there's a paper about it and you can search it online. So through this architecture, TIKV like scales to petabytes of data effortlessly and more importantly, all those scaling rebalancing works and it's all done by Taikevi itself automatically. So in case you're curious and if we zoom out the TIKV architecture a little bit, here's the TIDP architecture, the whole system. So as I mentioned before, TIDP is a query engine and sitting on top of the storage. And similarly, we also have another query engine called TI Spark. So you can write some Spark like commands or query using Spark syntax. And on the side, we have a component called PlacementDriver. That's sort of like a centralized service to store some of the metadata of our system. And, you know, there's a new thing in our system that is called TI Flash because TIKV is the raw, like the old, raw-based storage engine. So it's aiming for like OLTP workload and there are more and more customers like one OLTP system. And so we built a component for that scenario that is called TI Flash and it's in column-based data structure. So TIKV or TIDP can be deployed both on print and on cloud. You know, like actually many of our bank users, like big bank users, they prefer to deploy TIKV on their data center and TIKV on their data center. I mean, it's okay, but I have to say it's hard to support them because, like, when anything happens, we have to go to their place to do some work. It's not a skillable solution. Well, the good side of this is that they're very rich. You know, these big customers, they're very rich so they don't care about cost very much. They give us the hardware and we use it and they don't ask any questions about, like, do you have some room left? Is the resource enough on that? Well, but now our focus is cloud. So, you know, we're a cloud-native database. So, by the way, we are deployed on cloud using Kubernetes. I'm not sure how familiar are you with Kubernetes, but we use Kubernetes operator pattern to deploy our product. So, the good thing about cloud is it's a more skillable business, especially for ourselves. But, you know, when we provide our service as, like, software as a service on cloud, we have to care about, like, cost and resource and efficiency and all those things. You know, like, everything is not free on cloud, right? So, you have to pay for the computational resource, like virtual machine, EC2 instance, and you also need to pay for the storage. And, you know, like, for example, the EBS, you're not only paying for the space, you're also paying for the number of aisles and, like, a bandwidth, something like that. And, you know, if you have experienced building a service on cloud, you will know that there's, like, a price difference between cross-region traffic, cross-zone traffic, and network traffic. So, there's many things, like, specific on cloud. So, what we can do to reduce the cost or make the resource more efficient? So, here are some examples I can think of.  And the technical level, there are much more works we can do, like operational work. So, if we can reclaim or shrink down the unused resource, we save a lot of money. But, in this talk, in our talk today, I mainly want to focus on, like, architectural changes and optimizations we did, like, to our main component, which is a techie. So, on the right here, you can see, like, a resource analysis of TIDB or TIDB on cloud, like, 75% of the budget goes to, like, easy to instance, for example. So, how can we, like, make the resource more efficient, right? So, for computational resource, we can, in general, we can write, like, more efficient code. That's the most effective one, probably. But, we can also, like, do some work, like, reducing the unnecessary process. And for storage, we can, like, we can have a system with a smaller size. We can reduce the number of IOs or have, like, a storage service, storage service with lots, or with a smaller bandwidth. And for network, like I mentioned before, they have different, like, press difference. So, we want to reduce the long-distance traffic as much as we could. So, first, computational efficiency. As mentioned before, we use roughed algorithm for data replication, or, more specifically, multi-roughed. That means each roughed group stores a piece of data, and the whole dataset of TIDB is stored by many roughed groups. So, I'm sure that, you know, you can imagine inside each roughed group, all the peers, they have to communicate with each other to make sure that they are in sync, right? And this process is called a heartbeat. You don't need to know, like, more details about it, but I'm sure you can understand the heartbeat. So, well, if you go with naive implementation of roughed, you will find it not super efficient. Why? Because when the dataset, like, becoming large, and there will be, like, millions of roughed groups in your system. And, well, in reality, you know, like, most of the groups store code data, right? So, what we can do is really to putting those inactive groups into sleep. So, and we call them hibernated regions internally. So, by doing this, we were able to greatly save a lot of CPU resources and, you know, of course, some network as well. So that's the first thing we did, like optimizing the multi-rough process. Second, you know, I see many of you guys, like, use MacBook. So, probably running Apple M1 or M2 and I have a laptop which is running on Apple M2. So, you all know that the power of ARM, like, not new, but, like, the other CPU architecture. So, you know, like, we have been trying to migrate our service from, like, X86 to ARM and our testing shows that by using ARM we can save, like, around 20% of the budget but achieve the same level of performance. Of course, right? So, if the performance is not the same, it's pointless. That's about, like, computational resource and next is about storage efficiency. So, here is a graph showing you, like, the life of a write in Type-AV. So, we just talked about the hard bits, like, just now. And that's more like the rough core and this is more like the rough application. So, we can also divide the whole process into three steps. So, a request is coming to the leader of the rough group and the leader replicates the rough log to its followers and all the followers will, like, persist the rough log in their local storage engine and we call it, like, rough storage and then there will be another, like, process called log applying process and they will fast log from the local rough storage and do some processing and then put the real user data into the real user data storage engine. So, I guess, like, we can tell that there are mainly two types of IO here. One being the rough log and the other one being the real user data. So, we will discuss them separately. So, first, let's see what we can do here to the real user data step. So, you know, on each node, on single node, Type-AV uses ROXDB as a local storage engine. ROXDB is an LASM tree structure implementation and the LASM tree structure looks like this, roughly. So, a write coming to an LASM tree will be put into both the memory table, which sits in memory, and as well as the write-highlog, which sits in the disk. And, like, when the memory table goes beyond a certain limit, it will be flushed into the disk, in shortest string table, SSD. Well, you know, if the engine crashes, there will be some data in the memory table, and there will be loss, right? And that's the time when the write-highlog picks in. So, the recovery process will read the log and rebuild the memory table. So, this is like a typical LASM tree implementation. Well, in reality, you know, in our system, write-log actually plays the same role as write-highlog, like native write-highlog of ROXDB. So, in our case, you know, what we can do is, instead of using the native write-highlog of ROXDB, we can just use write-log. So, you know, this optimization is actually kind of like a no-brainer. You may ask me, like, why didn't we do this at the first place? I will tell you, like, two reasons, one truth and one lie. So, the first reason is, using ROXDB native write-highlog, we can get faster recovery time. And the second reason is, we don't have the manpower. You can guess which one is truth, which one is lie. Okay, that's about, like, a real user data step. And as for the write-log, you know, in the past, actually, we store both user data and write-log in the same ROXDB instance. So, you know, our SMTree or ROXDB, in general, they have, like, better write performance comparing to BTree. Well, our SMTree structure usually needs some background activity to do some work, you know, to make sure the read is being performed. And that's called compaction. And there's, like, a well-known problem of this process. It's called write amplification. Because, you know, you flash the data from the memory table to a sort of string table, and there will be, like, a lot of files set in the desk. And sometimes you have to merge them together, like, load them into your memory, and then merge them together, and write back to the disk. So, like, one single entry will be read into the disk multiple times because of this process. And that's called write amplification. And, you know, but one character ROXLog has is they are coming in straight sequential order. So, what that means is that we don't really need the merge functionality or the compaction functionality of our asymetry. In this case, so we wrote our own, like, a new storage engine. We call it rough-ending. Oops. Yep. So, through rough-ending, we were able to reduce the total number of I.O. by, like, 20 to 40%. That was, like, a huge win for us. Yep, that's about I.O. So, for a network, you know, like, as mentioned, cross-region traffic and cross-route traffic are significantly higher than local network traffic. So, you know, our works here are mostly about reducing the long distance traffic. So, TIKVN or TIDB is a distributed system. We usually deploy the whole system in different geographical areas and then scatter the replicas, like, one replica per zone. You know, if that's the case, if, like, by default, there's no much room we can improve this process because we have to replicate the data, right, at least once. But, again, in reality, some of the users, they have more than one replica in the same zone. A typical use case is, as mentioned, we have new component tap flash, sometimes they deploy a tap flash in addition to TIKVN in the same zone. So, in this case, you know, if we don't do anything about it, this two replica will, like, may try to gather own data from the leader, which is in another zone, right? So, obviously, we can optimize that. So, instead of, like, getting data from the leader at the same time, we can force the second replica in the same zone to gather data from the first one, right? Just copy the data across zones once. And similarly, the same principle can also be applied to the read flow as well. So, like, one sentence, like, read locally as much as possible. But this requires utilizing the follower read functionality and provided, which is provided by MVCC, which is, like, multi-version concurrency control plus wrapped algorithm. Here's, like, a summary. So, for computational resource, we optimize it by optimizing the multi-wrapped heartbeat flooding problem, and we're also able to reduce the cost by using ARM. And for storage, we write our own wrap log engine. It's also open source, and using that gives us, like, 10 to 40% less IO and some computational advantages as well because we don't need to do the connection. And we substitute ROXDB write-out log. That saves us a lot of IO as well. Then, for network, the main principle is copy data only once across zones. So, here are some future works we are planning to do. Basically, leveraging more cloud services. For example, Cloud Lambda. It's sort of, like, on-demand computing framework, so we can use that to do compaction. And secondly, we can use EBS snapshot functionality to do the backup. And last, you know, currently, we mainly use EBS, we don't use S3. S3 has, like, large throughput, but the performance is not good. But in the future, if we can distinguish code data from the hot data very accurately, we may use S3. Yep, that's it. That's our topic today. So, questions? Thank you, Randy. So, anyone has any questions? I can then walk around with a mic. Oh, sure. Thank you. No problem. Check. Here we go. So, for keeping traffic in the same AZ, if I were, say, deploying my own TI-KV, is there a way that I would need to configure that to do that? No, it's our work. Yeah, a little good, yeah. So it's just aware of which zone it's in? But I think there's one where we may need to cooperate, which is to label the server. Okay, yeah. So you tell it which zone the server's in, then thank you and it takes care of it. That's cool. Thank you. Anybody else? Thank you. I have two questions. First is to reduce the harbids of raft. Those harbids are for different consensus groups, and they have different, for example, terms. So how does it actually work? Like who are the sender and what information does it combine together and who are the receivers? I imagine that will be every leader or basically every node because it must be a leader or it's very likely to be a leader of some consensus group to send everyone else. Yes, exactly. So the picture I was drawing there was from a per-group perspective. Yeah, this is just one group. The leader sends harbids to his followers, right? So what we did was just to stop the harbid process if there's no rights into this group. But if there's a request, then we have to resume the whole rough process. Oh, it's not to piggyback a bunch of harbids into one message. It's more like reduce the harbids interval for the idle groups. Right. So what you're talking about is another optimization. Oh, got you. Thank you. I think that's it. Okay, sure. Anybody else? There, please. Sorry, I joined late. But how can you describe the architecture? If I need to kind of use this in my AWS tenancy, how would I use it? Yeah, so how do you use, like, technology? Explain the architecture. If I need to use your service right now in my tenancy, how would I use it? I think this is what you're looking for, this. And like I said before, so we're being deployed on cloud using Kubernetes, more specifically, Kubernetes operator pattern. Yeah, this is like the whole system. And by the way, we also have our own deployment tool called tie-up, T-I-U-P. If you want to play with it, like in your local environment, you can use that as well. But on cloud, we mainly use Kubernetes. Right, I would encourage you to use tie-up. And so it will actually even download the binary for you and copy it around everywhere and then start them all up and get them talking to each other. Yes, yes, yes. Thank you, another run. So to allow the type flash node to fetch the data from other followers, it's pretty different from the raft protocol where the leader sends all the messages to the followers in push manner. So I guess the type flash nodes can only fetch the committed data in the learner mode. Is that how it works? Yeah, exactly. It's a learner. Gotcha. So for followers, they have to get the message from the leader directly, so they cannot follow each other. Yeah, I mean, implement the patient detail-wise, there are many more works. Yeah, we have some other internal component called a rough proxy, right? Yeah, yeah, but that's like the work of it, yeah.  Thank you. But in general, you're right. It's a learner, yeah. Good question. Anybody else? That's it. Great talk, Andy. Thanks. Okay, thank you so much for coming here.