from our studios in the heart of Silicon Valley, Palo Alto, California, this is a CUBE Conversation. Hello everyone, well, this is a special CUBE Conversation and we are here in Palo Alto, California, the CUBE studios. Here at Tony, Gianna Domenico, who's the senior security strategist and research at Fortinet and Fort Agard Labs, live from Las Vegas, where Black Hat and then DEF CON security activities happening. Tony, also known as Tony G. Welcome to this CUBE Conversation. Hey, thanks, John, man. Thanks for having me. So a lot of action happening in Vegas. We just live there all the time with events. You're there on the ground. You guys have seen all the action there.  Got a copy of it right here with the threat index on it. Talk about this quarterly global threats report because the backdrop that we're living in today, obviously you're at the conference and the cutting edge is security, impacting businesses at such a level. We almost have shell shock from all the breaches and threats that are going on. Every day you hear another story, another story, another hack, more breaches. It's at an all-time high. Yeah, I think a lot of people start to get numb to the whole thing. It's almost like they're kind of throwing their hands up and say, oh, well, I just kind of give up. I don't know what else to do. But obviously there are a lot of different things that you can do to be able to make sure that you secure your cybersecurity program. So at least you minimize the risk of these particular breaches happening. But with that said, with the threat landscape report, what we typically do is we start out with this overall threat index. And we started this last year. And if we fast forward to where we are in this actual Q2 report, it's been one year now. And the bad news is that the threats are continuing to increase. They're getting more sophisticated. The evasion techniques are getting more advanced. And we've seen an uptake of about 4% in threat volume over the year before. Now, the silver lining is, I think we expected the threat volume to be much higher. So I think, though it is continuing to increase, I think the good news is it's probably not increasing as fast as we thought it was going to. Well, it's always, you have to know it. You're going to have to look for it. A lot of people talk about what you can't see. And there's a lot of a blind spot there. It's become a data problem. I just want to let people know they can find the report, go to Fortinet's website, there's a blog there for the details of all the threat index. But the notable point is it's only up 4% from the position year of the year. The attempts are more sophisticated. So I got to ask you, is there stuff that we're not seeing in there? I mean, is there blind spots? What's the net net of the current situation? Because observability is a hot topic in cloud computing, which is essentially monitoring 2.0. But you got to be able to see everything. Are we seeing everything? What's out there? Well, I mean, I think us as you know, Fortiguard and our cyber threat intelligence, I think we're seeing a good amount. But when you talk about visibility, if you go back down into the organizations, I think that's where there's definitely a gap there because a lot of the conversations that I have with organizations is they don't necessarily have all the visibility they need from cloud all the way down to the end point. So there are some times that you're not going to be able to catch certain things. Now with that said, if we go back to the report, at the end of the day, the adversaries have some challenges to be able to break into an organization. And of course, the obvious one is they have to be able to circumvent our security controls. And I think as a security community, we've gotten a lot better at being able to identify when the threat is coming into an organization. Now on the flip side though, if you refer back to the MITRE attack knowledge base, you'll see a specific tactic category called defense evasions. There's about 60 plus techniques, evasion techniques, the adversary has at their disposal, at least that we know, there may be others, but so they do have a lot of opportunity, a lot of different techniques to be able to leverage. With that said, there's one technique. It's a disabling security tools that we started seeing a bit of an increase in this last Q2 threat landscape report. So a lot of different types of threats in malware have the capability to be able to one, look at the different processes that may be running on a workstation, identifying which one of those processes happen to be security tools and then disabling them. Whether they're, maybe they might just be able to turn the actual service off, or maybe there's something in the registry that they can tweak that'll disable the actual security control. Maybe they'll actually suppress the alerts, whatever they can do to make sure that that security control doesn't prevent them from doing that malicious activity. Now, with that said, on the flip side, from an organization perspective, you wanna make sure that you're able to identify when someone's turning on and turning off those security control to any type of alert that might be coming out of that control. Also, and this is a big one, because a lot of organizations don't necessarily do this, minimize who has the ability to turn those particular security controls on and off. The worst case is you don't wanna have all of your employees, you don't wanna give them the ability to be able to turn those controls on and off.  I mean, this increase in malware and exploit activity you guys are pointing out is clearly a challenge. The other thing that the report kind of teases out, I wanna get your opinion on this, is that the upping the ante on the evasion tactics has been very big trend.  Talk about this feature of upping the ante on evasion tactics. Yeah, so that's what I was kind of referring to before with all the different types of evasion techniques, but what I will say is most of all the threats these days all have some type of evasion capabilities. A great example of this is every quarter, if you didn't know, we look at different types of actors and different types of threats and we find one that's interesting for us to dig into, and we'll create what's called an actual playbook where we wanna be able to dissect that particular threat or those threat actor methodologies and be able to determine what are their tactics and corresponding techniques, which sometimes, of course, includes evasion techniques. Now, the one that we focused on for this quarter was called Zegos. What Zegos is, it's a specific threat that is an information stealer, so it's gathering information really based on the mission goals of whatever that particular campaign is, and it's been around for a while going all the way back to 2011. Now, you might be asking yourself, well, why do we actually choose this? Well, there's a couple of different reasons. One happens to be the fact that we've seen an uptick in this activity, usually when we see that, it's something we wanna dive into a little bit more. Number two, though, this is a tactic of the adversary. What they'll do is they'll have their threat there for a little while and then they'll go dormant. They'll stop using that particular malware that's a specific sort of threat.  Security organizations, vendors might actually do the same, let that digital dust kind of settle, and then they'll come back bigger, faster, stronger. And that's exactly what Zegos did, is we looked at a specific campaign and this new malware, the new and improved malware, is they're adding in other capabilities for not just being able to siphon information from your machine, but they're also now can capture video from your webcam. Also, the evasion techniques, since we're on that particular subject, what they're also able to do is they're looking at your application logs, your system logs, your security logs, deleting them, making it a lot more difficult from a forensic perspective, be able to go back and figure out what happened, what that actual malware was doing on the machine. Another interesting one is they were looking at a specific JPEG file. So they were looking for that hash and if the hash was there, the actual malware would run and we didn't know what that was, so we researched a little bit more and what we found out was that JPEG file happened to be a desktop sort of picture for one of the sandboxes. So it knew if that particular JPEG was present, it wasn't going to run because it knew it was being analyzed in a sandbox. So that was a second interesting thing. The third one that really leaned us towards digging into this is a lot of the actual security community attribute this particular threat back to cyber criminals that are located in China. The specific campaign that we were focused on was on a government agency also in China. So that was kind of interesting. So you're continuing to see these these malwares maybe sort of go dormant for a little bit, but they always seem to come back bigger, faster, strong. And that's by design. This is that long haul, long view that these adversaries are taking in there, actually organized as economies behind what they're doing. They're targeting this, not just hit and run. It's get in, have a campaign. This long game is very much active. How do enterprises get on top of this? I mean, is it a people process issue? Is it some tech from ForteGuard Labs or what's Fortinet's view on this?  It's really, it's a combination of everything. It's a combination, you kind of hit like some of it. It's people, it's processes and technology. Of course, we have a people shortage of skilled resources, but that's a key part of it. You always need to have those skilled resources. Also making sure you have the right processes. How are you actually monitoring things? I know a lot of folks may not actually be monitoring all the things that they need to be monitoring from what is really happening out there on the internet today. So making sure you have clear visibility into your environment and you can understand and maybe give them point in time what your situational awareness is. From a technology perspective, you start to see, and this is kind of a trend, we're starting to leverage artificial intelligence, automation, the threats are coming and it's such a high volume. Once they hit the environment, instead of it taking hours for your incident response to be able to at least not necessarily mitigate but isolate or contain the breach, it takes a while. So if you start to leverage some artificial intelligence and automatic response where the security controls are working together, that's a big part of it. Awesome, thanks for coming on. This is a huge problem. I think no one can let their guard down these days. Certainly with the service here expanding, we're going to get to that talk track in a second. I want to quickly get your thoughts on ransomware. This continues to be a drum that keeps on beating from an attack standpoint. It's almost as if when the attackers need money, they just hit the same ransomware target again. They pay in Bitcoin. This has been kind of a real lucrative but persistent problem with ransomware. What's going on with ransomware? What's the state of the report and what's the state of the industry right now in solving that? Yeah, yeah. We alluded to this a little bit in last quarter and actually a few quarters. And this is a continuous sort of trend. Ransomware typically is where it's on the cyber crime ecosystem. And a lot of times the actual threat itself is being delivered through some type of a phishing email where you need a user to be able to click a link or click an attachment. And it's usually kind of a prey and spray thing. But what we're seeing is more of a targeted approach. What they'll do is they'll look for and do some reconnaissance on organizations that may not have the security posture that they really need to have. It's not as mature and they know that they might be able to get that particular ransomware payload in there undetected. So they do a little reconnaissance there. And some of the trend here that we're actually seeing is they're looking at externally RDP sessions. There's a lot of RDP sessions, the remote desktop protocol sessions that organizations have externally so they can enter into their environment. But these RDP sessions are basically not as secure as they need to be either weak username and passwords or they are vulnerable and haven't actually been passed and they're taking advantage of those. They're entering in there. And then once they have that initial access into the network, they spread their payload all throughout the environment and hold all those devices hostage for a specific ransom. Now, if you don't have the particular backup strategy to be able to get that ransomware out of there and get your information back on those machines again, sometimes you actually may be forced to pay that ransom. Not that I'm recommending that you sort of do so but you see organizations are deciding to go ahead and pay that ransom. And the more they do that, the more the adversary is going to say, hey, I'm coming back and I know I'm going to be able to get more and more. Yeah, cause they don't usually fix the problem or they come back in and it's like a blank check for them. They come in and keep on hitting the same target over and over again. We've seen that at hospitals, we've seen it at kind of the more anemic IT departments where they don't have the full guard capabilities there. Yeah, and I would add on, what's really becoming a big issue. You know, and I'll ask you a question here, John. I mean, what does Microsoft, NSA and DHS have in common for this last quarter? Robin Hood. That's actually a good guess. What they have in common is the fact that each one of them urge the public to patch a new vulnerability that was just released on the RDP sessions called Blue Keep.  You didn't really need to have a user click a link or click an attachment. You know, basically when you would actually exploit that vulnerability, it could spread like wildfire and that's what wormable is. A great example of that is with WannaCry a couple years ago, it spread so quickly. So everybody was really focused on making sure that that vulnerability actually gets patched. Adding on to that, we did a little bit of research on our own and ran some internet scans and there's about 800,000 different devices that are vulnerable to that particular new vulnerability that was announced. And, you know, I still think a lot of people haven't actually patched all of that. And that's a real big, you know, concern especially because of the trend that we just talked about ransomware payload, the threat actors are looking at our RDP as the initial access into the environment. So on Blue Keep, that's the one you were talking about, right? So what is the status of that? You said there's a lot of vulnerabilities out there. Are people patching it? Is it moving down the path in terms of are people on it? What's your take on that? What's the assessment? Yeah, so I think some people are starting to patch but shoot, so the scans that we do, there's still a lot of unpatched systems out there. And I would also say we're not seeing what's inside the network. There may be other RDP sessions in the environment inside an organization's environment which really means now if ransomware happens to get in there that has that capability then to be able to spread like via some RDP vulnerability, that's gonna be even a lot more difficult to be able to stop that once it's inside a network. I mean, some of the recommendations obviously for this one is you want to be able to patch your RDP sessions for one. Also, if you want to be able to enable network authentication, that's really going to help as well. Now, I would also say maybe you want to harden your username and passwords but if you can't do some of this stuff at least put some mitigating controls in place. Maybe you can isolate some of those particular systems, limit the amount of access organizations have or their employees have to that or maybe even just totally isolated if it's possible. Internal network segmentation is a big part of making sure you're able to mitigate some of these potential risks or at least minimize the damage that they may cause. Tony, G, I want to get your thoughts on your opinion and analysis expert opinion on the attack surface area with digital and then ultimately what companies can do for. Well, let's start with the surface area. What's your analysis there? A lot of companies are recognizing obviously with IoT and other digital devices, the surface area is just everywhere, right? So, you know, gone to the perimeter days, that's kind of well-known. It's out there. What's the current digital surface area threats look like? What's your opinion? Sure, yeah, yeah, it's funny these days. You know, John, I'll tell you, I'd say everything that seems to be made as an IP address on it, which means it's actually able to access the internet and if it can access the internet, the bad guys can probably reach out and touch it and that's really the crux of the problem these days. So, anything that is being created is out on the internet and like we all know, there's really not a really rigged security process to make sure that that particular device as secure is as secure as it actually needs to be. Now, we talked earlier on about, you know, IoT as it relates to maybe home routers and how you need to be able to harden that because you will see a lot of IoT botnets that are taking over those home routers and creating these super large IoT botnets. On the other side of it, you know, we've seen a lot of SCADA systems now that traditionally were in air-gapped environments. Now they're being brought in to the traditional network, they're being connected there. So there's an issue there, but one of the ones we haven't actually talked a lot about and we're starting to see the adversaries focus on these little bit more as devices in smart homes and smart buildings. In this Q2 Threat Landscare Report, there was a vulnerability in one of these U-motion business management systems. And, you know, we looked at all the different exploits out there and the adversaries were actually looking at targeting that specific exploit on that smart management building service device. We had about 1% of all of our exploit hits on that device. Now that might not seem like a lot, but in the grand scheme of things, when we're collecting billions and billions of events, it's a fairly substantial amount. Now that really starts to kind of bring a whole another thought process into, as a security professional, as someone responsible for securing my cyber assets. What do I include in my cyber assets now? Do I include all the business management systems that my employees are in for my overall business? Now that actually might be connected to my internal network where all of my other cyber assets are. Maybe it actually should be, maybe it should be part of your vulnerability patch management process, but what about all the devices in your smart home now? All these different things are available and you know what the trend is, John, right? I mean, the actual trend is to work from home. So you have a lot of your remote workers have a great access into the environment. Now there's a great conduit for the adversaries to be able to break into some of those smart home devices and maybe that from there, they're on the employee's machine and that kind of gets them into the other environment. So I would say start looking at, maybe you don't want to have those home devices as part of what you're responsible for protecting, but you definitely want to make sure your remote users have a hardened access into the environment. They're separated from all of those other smart home devices and educate your employees on that. And the user awareness training programs talk to them about what's happening out there, how the adversaries are starting to compromise or at least focus on some of them smart devices in their home environment. These entry points, or you can point out are just so pervasive. You have work at home, you're totally right. That's a great trend that a lot of companies are going to and this is a virtual first come in the world, we build this new generation of workers they want to work at home anywhere.  I know, I know, gosh, this kind of concern me, you know, say for example, and then what's hot these days is the webcam, right? You know, let's say you have an animal and you happen to go away, you always want to know what your animal is doing, right? So you have these webcams here. I bet you, someone might be placing a webcam that might be near where they actually sit down and work on their computer. Someone compromises that webcam, you maybe they can see some of the username and passwords that you're using to log in, maybe they can see some information that might be sensitive on your computer.  Tony, Gia, I want to get your thoughts on how companies protect themselves because this is the real threat. An IoT doesn't help either industrial IoT to just internet of things, whether it's humans working at home to, you know, sensors and light bulbs inside other factory floors or whatever, I mean, it's everywhere now. The surface area is anything with an IP address and power and connectivity. How do companies protect themselves? What's the playbook? What's coming out of Red Hat? What's coming out of Fortinet? What are you advising? What's the playbook? Yeah, you know, I, you know, when I get asked this question a lot, I really, I sound like a broken record sometimes and I try to find so many different ways to spin it. You know, maybe I can actually kind of say it like this and it's always means the same thing. Work on the fundamentals. John, you mentioned it earlier from the very beginning, visibility, visibility, visibility. If you can't understand all the assets that you're protecting within your environment, it's game over from the beginning.  If you're not aware of what you're actually protecting, there's just no way that you're gonna be able to understand what threats are happening in and out of your network. At a higher level, it's all about situational awareness. I want to make sure if I'm, if I'm a CISO, I want my security operations team to have situational awareness at any given moment all over the environment, right? So that's one, you know, grabbing that overall sort of visibility. And then once you can understand where all your assets are, what type of informations on those assets, you get a good idea of what your vulnerabilities are. You start monitoring that stuff. You can also start understanding some of the different types of gaps. I know it's challenging because you got everything in the cloud all the way down to the end point, all these mobile devices. It's not easy, but I think if you focus on that a little bit more, it's gonna go a longer way. And I also mentioned, we as humans, when something happens into the environment, we can only act so fast. And I kind of alluded to this earlier on in this interview where we need to make sure that we're leveraging automation, artificial intelligence to help us be able to determine when threats happen, you know, to actually be in the environment, being able to determine some anomalous activity and taking action. It may not be able to remediate, but at least it can take some initial action. The security controls can talk to each other, isolate the particular threat and let you fight to the attack, give you more time to figure out what's going on. If you can reduce the amount of time it takes you to identify the threat and isolate it, the better chances that you're going to have to be able to minimize the overall impact of that particular region. You know, Tony, you're jogging up a lot of memories from interviews I've had in the past. I've interviewed some four-star generals, head of NSA, head of cyber command. You get a lot of military kind of thinkers behind the security practice because there is a keep an eyes on the enemy, on the target, on the adversary kind of dialogue going on. They all talk about automation and augmenting the human piece of it, which is making sure that you have as much real-time information as possible so you can keep your eyes on the targets and understand to your point contextual awareness. This seems to be the biggest problem that CSOs are focused on, how to eliminate the tasks that take the eyes off the targets and keep the situation awareness on point. Your thoughts on that? Yeah, yeah, I have to, you know what's, I used to do, and I still do them now, I do a lot of presentations about situational awareness and being able to build your security operation center and to get that visibility. And I always start off with the question of, when your CISO walks in and says, hey, I saw something in the news about a specific threat, how are we able to deal with that? 95% of the responses are, well, I'd have to kind of go back and kind of like, you know, I'd have to actually come dig in and see and it takes them a while for them to be able to get, right? Yeah. So yeah, the classic was, let me get back to you, boss. What a patch, patch that thing. Tony G, thank you so much for the insight. Great, congratulations on the quarterly report. Keep up the good work. Quick, quick story on Black Hat. What's the vibe in Vegas? DEF CON is right around the corner after it. You're seeing the security industry become much more broader. Obviously as the industry surface area becomes from technical to business impact, you're starting to see the industry change. Amazon Web Services had an event, cloud security called Reinforce. You're starting to see a much broader scope to the industry. What's the big news coming out of Black Hat? Yeah, you know, it's a lot of the same. You're the thing that actually kind of changes. There's just so many different vendors that are coming in with different types of security solutions and that's awesome. That is really good. With that said though, you know, we talked about the security shortage that we don't have a lot of, you know, security professionals with the right skill sets. What ends up happening is, you know, these folks that may not have that particular skill, you know, needed, they're being placed in these higher level security positions and they're coming to these events and they're overwhelmed. Cause they're all, they all have a slight, it's all a similar message, but slightly different. So how do they determine which one is actually better than the other? So it's, I would say from that side, it gets to be a little bit kind of challenging, but at the same time though, I mean, we continue to advance. I mean, from the, you know, from the actual technical controls, solutions perspective, you know, we talked about it. You know, we're getting better with automation, doing the things that the humans used to do, automating that a little bit more, letting technology do some of that mundane, everyday kind of grind activities that we would, as humans would do, would take us a little bit longer, push that off, let the actual technology controls deal with that so that you can focus, like you had mentioned before, on those higher level, you know, issues and also the overall sort of strategy on either how to actually not allow the adversary to come in or how to determine once they're in and how quickly are we able to get them out? You know, we talk, we have a panel of CISOs that we talked to and we were running surveys through them, through theCUBE, insights. Most CISOs we talked to, obviously they all want to talk off the record and they don't want anyone to know what they work for, they all talk to them, they say, look, I'm bombarded with more and more security solutions. I'm actually trying to reduce the number of suppliers and increase the number of partners. And this is nuanced point, but to your, what you're getting at is a tsunami of new things, new threats, new solutions, that could be either features or platforms or tools, whatever, but most CISOs want to build an engineering team, they want to have full stack developers on site, they want to have compliance teams, investigative teams, situational awareness teams, and they want to partner with suppliers. They want partners, not just suppliers. So reduce the number of suppliers, increase the partners. What's your take on that?  Do you agree with that statement? Well, that's true. Yeah, yeah, I mean, that's actually really our whole strategy, overall strategy for Fortinet is, and that's why we came up with this security fabric. We know that skills are really not as prevalent as they actually need to be. And of course, there's not endless amounts of money as well, right? And you want to be able to get these particular security controls to talk to each other. And this is why we built this security fabric. We want to make sure that the controls that we're actually going to build in, and we have quite a few different types of security controls that work together to give you the visibility that you're really looking for. And then here's a trusted partner that you can actually kind of come to, and we can work with you on, one, identifying the different types of ways the adversaries are moving into the environment and ensuring that we have security controls in place to be able to thwart that threat actor playbook, making sure that we have a defensive playbook that aligns with those actual TTPs in the offensive playbook. And we can actually either detect or ultimately protect against that malicious activity. Tony, thanks for sharing your insights here on theCUBE conversation. We're going to have to come back to you on some of these follow-on conversations. We'd love to get your thoughts on observability, visibility, and get into what kind of platforms are needed to go this next generation with cloud security and surface area being so massive. So thanks for spending the time. Appreciate it. Hey, thanks a lot. All right. Have a great time in Vegas. This is theCUBE conversation. I'm John Furrier here in Palo Alto. Talking with Tony G with Fortinet in Las Vegas. Thanks for watching.


On today's agenda, we've got our usual hackfest planning session, an update on the internship program, which is now in the phase of getting some interns. We don't have any quarterly updates for the working groups yet. This week, I should say, but we do have one for borough, and I believe that's ready to go. Yes, it is. Okay. And then we have the continued discussion of going to 1.0 before exiting incubation. So any other topics for today's discussion? Okay. No. Talag, do you want to kick it off? Sure thing. All right. So for the hackfest, I think two things we wanted to discuss today. Let's maybe tackle the easier one first, which is Dubai. This came up at the hackfest in LA as well as on the call last week. The sense we're getting is that with the holiday in Europe and just kind of general travel availability, it's sounding like many of the folks from the more seasoned community are not able to travel for this. So it's sounding as though we may just want to turn this into a meetup as opposed to a formal hackfest, but certainly want to close that discussion off with those on this call. Get any further input or thoughts on that? Doesn't sound like it. So maybe we should just have a vote. Quick. I guess, are there any objections if we just turn that into a meetup at the end of April for those that are in town, as opposed to doing a formal hackfest? Give the answer. All right. Sounds good. And then the second part of the discussion is just kind of around the format of the event. If anyone had further thoughts there, I think a couple of people made the good point that it sounds like there's really two parallel things happening. One is more focused on bringing new devs up their learning curve, and the other one is really having the more seasoned community focused on cross-project collaboration, hacking, et cetera. It may make sense to keep the format doing it consecutively, or it may make sense to put this off entirely into two unique events. So curious if there's been any more thoughts from those on the call related to that. I think what we probably need to do is, I mean, because I think that having some sort of a road show and maybe have a few key individuals join that, and maybe it's a regional kind of a thing with hitting up a few cities and doing a meet up a night or something like that for three consecutive nights or something like that, that that might be the approach to take, and they can be planned in advance, and maybe we have one of those a quarter, I don't know, or maybe twice a year we do that. And then, and then just have quarterly access. It seems to be the right thing. I just don't know, I mean, it doesn't sound to me like we would be doing the planning. I think that's more of a marketing thing, probably. Is that what the thinking is? Probably a combination of a few different functions. Yeah. Yeah. I agree with Chris here, you know, it's hard for the new people to really catch on and do any meaningful work at a lot of these hackfests. And when we host them in places we haven't been before, we tend to get a lot fewer, say, developers who really know what they're doing. So it might be best to do, I mean, I really, in agreement with Chris here, that maybe we do, we do some core hackfests in places where we have a lot of developers quarterly. And I think I've suggested earlier like one in East Asia, one in San Francisco, one in New York, and then one in like Amsterdam or London. So if we do that quarterly, we hit most of our developers.  All right. And it's not to say that we couldn't do a meetup in the city that we're in for a hackfest. I mean, I think actually that might be a good idea, but, you know, keep it separate. Keep the functions separate. So kind of like what we did in, I guess that was Chicago, right, where we had the meetup. Yep. So in the case of Amsterdam, since we already have that one announced and people registering, does it make sense to still keep a training day, but make it very clear that the training day is a one day event. And, you know, we just leveraged some of the people that are traveling in anyway and focus on that and really make it clear that the following two days are the more advanced hackfest cross-project collaboration, hacking, and have a bit more structured of an agenda there. And while, although it's three days in a row, we really treat them as two separate events, or does it make sense to just chop off one day? Yeah. I mean, if everyone's going to be there already, it makes sense to, you know, to have whoever would need to have come early and talk about evangelism. Yeah. You know, Amsterdam, we've been there before. It's actually pretty active from a hyperledger perspective already, you know, with ABN Amro and some others, you know, really sort of taking a large role there. I wonder how, you know, how necessary a day zero is really going to be. I wonder if we could just maybe reach out to some of the, if you will, the locals to get a sense for whether they would think that's a needed function or whether we could just do a meetup. Okay. So in general, I mean, I'm a bit concerned with this idea of splitting the events entirely for the reason that, so first, I don't know that we'll be able to attract really people to come to a separate event just for evangelization, or if it's any different from all the other things we already do, we have meetups and stuff like this going on. And then, you know, it just falls into that program. And the other part is, I mean, the reason we came up with this day zero idea was because we couldn't stop people from coming during our hack fest, right? And then we felt this was disruptive to the primary goal of the hack fest. And we say, okay, let's have a separate day just for that. And now, yeah, just because we say, okay, we want to have it, are we going to stop people from showing up and having this kind of expectation we're going to introduce them to all the technology? I don't know.  I just don't know that we are not falling back into what we had before. And then we haven't solved anything. So, I mean, I actually think three days is good. But I do get your, I do, I do take your point. I think, though, that by making a little bit clearer what we're trying to achieve, you know, and, you know, if you're going to have a day zero and it's a completely different thing. And then we make it very clear that the hack fest or the face to face or whatever we wanted to call it was really intended for those that are already immersed and, you know, project conversations or project to project conversations that are probably going to be above a lot of new people's heads that, you know, that we, just by being a little bit clear about how we advertise and market this thing, we may sort of, you know, do ourselves a favor, you know, the challenges have been in the past that we just have a hack fest and we have an introduction. And then you have somebody asking everybody else a lot of questions. And so that can be a little bit of distraction from actually having the kind of conversations and work that we, I know we all want to do. The other thing for keeping three days is I think that then we would have more opportunity to actually put the working groups in and we could, instead of doing tracks, we could actually have, you know, identity and architecture and, you know, white paper or something like that where others could participate and we wouldn't have to pick and choose. It just gives us a little bit more time to have those kinds of conversations. And I think that for many in the community that are participating primarily through the working groups that that would be good and I think it would also be good for the people that are primarily working on writing code to also participate in those. So I'm actually in favor of keeping three days but making it a little bit more whole in terms of with the whole of what we're all doing, you know, so having, you know, actual working group meetings where, you know, others can participate having the, you know, and then having enough time for the conversations and cross-project collaborations. That sounds good to me. The other thing is, you know, we only had one instance of a three-day meeting so far and, you know, granted there was a fairly low attendance but that was across the whole meeting, right? And if anything, the day zero I couldn't attend it but it sounded like it was reasonably successful. Right, day zero was packed. So, you know, okay, so. Right, we had the high school students and we had a bunch of college students that showed up for the initial training who then didn't show up for day two and three. And I don't know if that's because they thought it was bad but I think it was mostly because of scheduling they had to go back to class. Right, so that's my point. I, you know, I don't think we should necessarily question the whole premise of the three-day meeting just because in the LA for some reason we didn't have too many people showing up especially for day two or one and two, so. So, this is the thing, this is the thing, Alec, can I just add something? Because when I speak to new members, I do spend a lot of time telling them about the hack fest and we all do and we tell them that day one is really a great onboarding for their dev teams and that they, you know, they don't need to participate in day two and three. I don't know how to say it, you know, day two and three gets much deeper. So, yeah, just, that is the way and maybe it goes back to Chris's comment. Maybe we need to market it better as we, you know, on the web page and when we talk to people about it but that's how I've been introducing it to folks. Yeah, I'm not sure how much long-term traction we get out of the day zero type visitors. I think that I'm interested in more of this deeper collaboration and working group progress that we get if we show up with a structured schedule of this is what we're going to get done and we don't stop people or discourage people from showing up but at least it's clear from the agenda that there's topics here that require context. So, there's probably some investment that needs to be made beforehand if you're going to show up to one of these. You know, the other thing that we could do, Dan, on that point is we could actually, I mean this is a little bit extra work but whoever it is that normally that would be, maybe doing a day zero presentation of the projects could just record a short video and we could have those posted the week prior, advertise them around the registration, for instance. Here's a set of introductory videos that you can go through and, you know, on your own time to give people some of that context and then allowing us to have the full time for our own conversations. Just a thought. So, I suggested this in chat but I think maybe if we had a separate registration process for the day zero and the actual hack fest it might serve as, you know, an effective filter. Yeah. Well, like I said, I thought I would go back to Tim and some of the other guys there and I'm sure I don't know who you're working with. Yeah, Yuri, Tim and a few others. Yeah. And then I would just get their thoughts. Okay. We have enough time to plan it. All right, sounds good. And then one other quick question and then I think let's wrap this up just because we have other topics. In terms of a quarterly cadence, you know, with Dubai not happening, is there a hunger to try to get either a US-based hack fest or more likely an Asia-based hack fest slotted in sometime from now until the end of June when we're in Amsterdam? It's common. I didn't catch that, Chris. Well, because I'm assuming that a lot of people are going to be in that consensus. Maybe I'm wrong. So, Chris, I think we lost most of what you said. We heard it's common. Yeah. All of you guys, I'm in a hotel and I, right, I'll put it in the chat but basically I'm suggesting we might want to do maybe one day around consensus. Yeah, we heard that. All right, if there's no other thoughts on this, we can take it off an email thread. Otherwise, I know we've got some other stuff to get through today. Yep. Okay, so let's jump to borough, I guess. Oh, no, internship program. Sorry. Yeah, just really quickly on the internship program, 12 projects. We're accepting applications from students now. Please help us promote that in the agenda that went out. There's a link to the blog with more details as well as some links. If you want to click to tweet and promote to your communities. All right. Super. And so next up would be borough Silas on. Hi, sorry, I managed to get into a pen drawing mode. Hi, everyone. This is Silas. So I've posted the update. If you want to be able to follow this as I go. There's also other couple of documents I've done. One being a roadmap post Morton. Which I might refer to of the Q1 roadmap that I did for the last update. And the other being a draft roadmap for Q2 that I'll work through the updates as it stands. So, yeah, in terms of project health, my feeling is that borough has been gaining momentum in terms of useful contributions on the Hyperledger chat in particular. Few users that I'll mention later who have done some useful stuff, including building a truffle like package for borough package manager facility code. The refactor was fairly big undertaking. It meant refactoring not only the borough code, but also getting all of our tooling into a into a mono repo called Boz Mama. But in the process, a lot of existing bugs were dealt with, including a lot of panics, a lot of type safety issues. But still, not huge progress on actual features, which was expected, but I kind of hope to close it off a bit quicker. So the main issues that we have, and some of these have a good chance of being closed off in what remains of Q1. So one is EVM compatibility. So I got a developer who's now up to speed from started consulting, who's been working on these EVM issues and has an implementation waiting in the wings for the revert op code. But there's a number of new op codes. And we're now in a good position to implement them. What I'm also doing at the moment is integrating the Ethereum has some table-based JSON tests available to all Ethereum clients, some of which can be translated to borough. So I wanted to integrate those to help us assert the correctness of the upgrades we do have. So for example, I found a bug with the help of one of the users who came on the Hyperledger chat in our sign extension, which basically meant that using types narrower than 256 bits that were positive was actually broken on this sign extension. It didn't set the sign bit correctly, which is kind of a fundamental thing to find, kind of strengthens the case for needing this level of testing. Another issue we've had quite a bit of, which pulls me in multiple directions is the lack of decent up-to-date documentation. We have some kind of extensive tutorials from the previous iteration of Monax, the company, which I point people towards. But there's a lot of that as atrophied where it sits, or there's just a lack of sort of basic information about how is our permission system defined. It's not complicated if you read the code, but I actually did add some comments on that. To that end, there are three more junior developers from TCS who have worked on a package of documentation, which due to some kind of arcane legal processes, I haven't actually been able to see. But hopefully that will give a, I spent several hours presenting burrow to them and asked them to put that into an overall kind of architecture piece of documentation. We also have a former employee from Monax who was deep in the tooling for years, who is working on consolidating a lot of our documentation. So I hope that will help it, but on the other hand, it's something that I could certainly do with help with because there are certain Monax priorities that involve burrow that are kind of trumping, spending time on documenting what's already there. And the other big one, and this relates to what I bring up in the Q1 post-mortem would be Web3 support. Partly, we internally re-prioritize that. We are taking burrow into production using our existing JavaScript APIs. There's also some issues with, we had been hoping to collaborate with people from Fabric EVM on a Go-based Web3 RPC, but they've decided to take a different approach. So that's something that hasn't happened, that was planned for community, then I think having Web3 support is quite a big deal for adoption, but like I say, for my day job, it wasn't really in the high priority set of things to work on with burrow, but for Hyperledger, it would be really useful if we could get some movement on that within burrow. There is a prototype or more than a prototype, I guess an alpha or beta Web3 implementation on Seth Sawtooth with burrow, but that's written in Rust and we wanted one in Go. So there hasn't been a release yet. There's quite likely since the last update, there's quite likely to be one this month, I would say, we're closing in on it, largely that's been down to the refactor, as I already mentioned, we had 0, 18, 0 would be the release, even though really it preserves a major point release, we're reserving at one because of the semantics that might be a wrongly have become attached to that. The other thing I'd like to start doing once that point release is out is integrating both in Fabric EVM and Sawtooth, the new changes, which are numerous and worth having, particularly in terms of EVM compatibility, that allows us to use the latest Solidity compiler, 0, 4, 2, 1. So activity, so this big refactor named Harkamama, that successfully completed, introduced a lot of safety, removed a lot of panics, mostly just really needed doing for sanity and maintainability. What ended up, I ended up doing in a, probably to a higher level than I had expected, but also took a bit more time was the establishment of this Mono repo, so in reality, to be able to test, borrow, and successfully, and to use it, all of our tooling that we would, we had slightly optimistically thought we could deprecate, has had to stick around. So we now have a Mono repo that has our compiler service, our key signing theme, and our JavaScript libraries, and our package manager, and integration tests in it, and there is now a sort of, kind of, dependency between borough and Bosmarmot, which is the name of this Mono repo for testing. So they both run a, quite a full set of integration level tests on any pool request, so that certainly gives a bit more certainty. The upgrade to 10-minutes, zero, 15-zero, that has loads and loads of improvements. There's actually a zero, 16, bit of a moving target with 10-minutes, but they're moving towards their 1.0 release for their Cosmos network, and we should be able to track them fairly quickly. We've also pushed various changes to 10-minutes around signing, because we run 10-minutes within a single process, and we need a bit more control over how we configure the node. Yeah, removal of fairly large 10s of different panics, it's a bit of a panic happy code base in some places. And the event system to synchronize 10-minutes was replaced in areas where there were blocking on events and so on was removed. So really heavily code quality improvements with some feature level things around. There's now a query language for events and logging, all of which were needed for debugging. So I'm just going to flick over to the post mortem just to see if there's anything to mention there. Yeah, I think I can leave that offline. So the current plans, and these are being driven as well by our use of borough for our product, which will be going into production Q2, Q3 this year. So a governance transaction, which will be, initially it will be something of a pointy stick that will allow you to redistribute validator voting power relating to the consensus mechanism and the native token across all accounts. Ultimately, we will want this under control of a governance, autonomous governance contract, but it will be building towards that. And that will also be reintroducing validator set changes, which have been commented out in an earlier form of validator bonding for a while. We have another team member joining who has a lot of database experience and will be working on upgrading our Merkle tree to something that Tenement have been working on and providing their SDK, but needs a little bit of work, but has some very interesting types of range proofs and other Merkle tree based proofs that it can provide, which we're going to want for running borough in this sort of multiple chain universe. State history, so that will be linked. We'll be able to have a queryable EVM history at any block, which should be a nice feature, something that Go Ethereum won't have in quite the same form. The event fire hose, this is the ability to get a catch or consumer for both consensus events and EVM events, which are defined at the contract level and are the way that Solidity drives a lot of external processes. Currently, you subscribe to that over WebSocket, but if you lose an event, there's no easy way to look back at previous ones. So this can give an experience that maybe feels a bit more like Kafka or something like that. The queryability events there. Validator bonding, it's linked, so we need a way of changing validator sets. And then a serialization of state and restoring that state. So being able to ship our state back ends around to different ephemeral nodes and Kubernetes is kind of what that's about, but also just giving an upgrade path. So if we can have a reasonably stable state store that we can dump to, we'd upgrade that to a chain that could read that in. On maintainer diversity, so Tyler Jackson from Monax, one of the founders of Monax has been getting more involved and I would consider him a maintainer now of Burrow. Me and Casey are still going. We have this new team member joining Sean, who will most likely be a maintainer and then there's another Sean from TCS who is keen to become a maintainer and they're kind of spinning up. I don't know, perhaps I could have some guidance on what level of confidence with the code base we should be waiting for before calling someone a maintainer. I'm fairly relaxed on how we do that, but perhaps wanted to give people a bit more time to get up to speed. In terms of contributors, and I'd like to think some of these people will be potential maintainers as well. Sean Blocker from TCS already mentioned. Amad Pulazade has done something quite interesting called SNAP, which is similar to Serium Truffle, but using our JavaScript APIs. He just came on to the chat and had this thing, so I've been trying to get his interest in perhaps maintaining our JavaScript libraries as part of Bosmama, whether they end up being an interim thing or turn into their own, stick around as an interface to grow that's so useful, I'm not sure, if Web 3 is around. Robert D. Bills has been building Kubernetes integrations. Early on, Adam Ludwig from Sawtooth helped me merge the large hypermarmor refactor, helped me review it. Some other stuff I think I mentioned there. The final thing to say is that we will be putting our own product into production on top of Borrow, which is our legal agreements network in Q2 and Q3. This both ups the ante for us in terms of testing around certain things like EVM compatibility and correctness of things like validator set changes and so on. But it also means that the focus is really there for us and things like documentation, possibly things like what we had hoped to prioritize around Web 3. They matter less for us now. I'm very interested in getting people from the community to work on that stuff. But that's just, I guess, the way the priorities lie with Monax. So yeah, I think that's all. Any questions from anyone on the Borrow update? Yeah, because I think your line is dropping out. Any questions from anyone on the call for Silas around the Hyperledger Borrow update? Hey Silas, this is Dan. That's a really thorough update. I appreciate you getting in such detail there. I was curious, did you make a giant merge to make that 666? Yeah, I noticed that. I didn't. It was not an earthly hand that did that. I guess Borrow already has a subterranean feel. That's what you will. More substantive question. So it's not like you've got one external repository that you're relying on for your testing. Everything else though is in the Borrow repo. Yeah, so we had certainly flirted with dropping all of our stuff, much of which we could Apache license into Borrow. There are two reasons, one hard reason and one softer reason that we haven't done it. The harder reason is that we still haven't done it. So what was Monax packages do, which is part of the Monax tool, I've stripped all of the docker dependencies and a lot of the kind of incidental complexity of that out and I've just kept the package manager. The package manager is a bit more than that. It reads these EPM.yaml files which can deploy contracts, it can call functions, and various things. It's what we use for integration tests, it's what we use to push contracts. That relies on Go Ethereum's ABI. It would be a non-trivial but not particularly excessive piece of work to get an Apache licensed ABI, but I am not familiar with them. So the ABI is the application binary interface for the EVM, you can call a function and you want to know how to pack the bytes for a particular function call and you want to know how to derive the dispatch hash for that function and all that sort of stuff, it's in the ABI. It would be nice to have an Apache licensed ABI. So that keeps part of what's in BOSMAM up there. I've actually dropped some GPL dependencies on things like Monax keys, which I think there is a lot of work to be done, particularly as Burrow takes on the ability to be a delegate for other keys that are within this Monax keys container. That may well move to Burrow. But the main idea was to get all of this stuff that you still essentially need to use Burrow into a single repo and to have it maintained. So we lost our person who had been working on it and we lost various people working on compilers and so on. So I am taking on maintaining that and BOSMAM up is a package that is sort of maintainable size and I feel like I can kind of iterate on. It's not entirely clear at this stage what it will become, but it's kind of essential really for someone getting into Burrow right now as we transition to the new code base. Okay, thanks. And then I imagine Adam probably has a good idea what's involved since he did review your big refactor. But I know we're all interested in getting all the benefits of that refactor integrated into Seth properly so we're not working off of that fork. So any guidance that you can suggest to Adam and probably defer first to Adam to see what we need to understand. We're looking forward to. So I think the thing with that will be probably to I need to get out of this 0180 release just to give a little bit of a rubber stamp to the there and then it would be best if we had Seth depend on Burrow as an actual dependency. And I'd note that if we use Go's depth tool, which is the first dependency management to go that actually seems to work, it will strip out unnecessary files. So you should be able to take just the Burrow components and they're all encapsulated in a bit of a better way now as well. But there are some critical bug fixes in there. So you guys will want it. Great. Any other questions, comments for Silas. Thank you. Got a lot of background noise might want to go please. Okay, so up next is continuing the discussion on whether a project can go to 1.0 before it leaves incubation or what are the implications of that. I think where we had left it off and can somebody paste in that link there. So we had continuing discussion. I think Dan, your point was that you think it is required. You seemed to be sort of required. I don't know if anybody else had any thoughts. Personally, I tend to think that when you're talking about 1.0, the code is either mature or it's not. But I think I can't remember who it was, maybe it was Hart was talking about, are we disadvantaging some of the non-core platforms, if they have a small community, if they have a robust and mature code base. Is that any less robust and mature? Obviously from a, you know, building a community, getting other, you know, getting committed diversity, getting maintainer diversity. Some of those other things may be harder to achieve with a smaller project. And I don't know if you can see where a project could say, well, we think we're at 1.0, but from a hyperlinked foundation, you know, supporting marketing, supporting with the, you know, security scan and, and both bounty and all the other things that go with it, including press releases and so forth, that that wouldn't necessarily be as, you know, if that were the case, but, you know, again, I'm, I'm just sort of coming at it from the perspective that says, you know, I think that we definitely want to encourage people to get out of incubation quickly. But that can be hard. And I think we all are aware that you know, we need to all do better. I think on growing that diversity and looking down some of the, you know, the closer roles around our various project. That's it's clearly something we need to be focusing on. But I don't know if that necessarily means we penalize the project.  It's kind of, it's something I spoke into Casey, our CEO about and would have a perspective on. So if you take, perhaps take borrower as a case study. So we have some of the the majority of the best practice is bad. That's actually one thing that the last quarter as well. And in terms of our code base age, having our first commits back in like, I think December 2014, we're actually older than than all of the, well, possibly all certainly sort of fabric projects. So in pure age, which is not I think quite what is meant by maturity. We're older, but we've also gone through some, some gymnastics and I wouldn't with the stuff that I've worked on recently want to put burrow into 1.0 yet. However, I may want to do that a bit sooner. And with us still being an incubation, I guess I actually wasn't even aware that that was potentially impediment. So I guess it depends how hard it is to get out of incubation. But certainly I would at some point feel like I'd want to give the signaling of a 1.0. I mean, I also would just like to just get on to a rapid release schedule and actually use major releases in the sort of semantic versioning. Meaning in which case, I mean, I nearly did this six months ago, to be honest, to get on to you know, if it was a major release, breaking changes as a lot of the releases are to just do it. That's a perspective from burrow. So Chris, I mean, you mentioned, there's like this seems like there is no clear there's no clarity on whether the current status of things is whether there is any requirement or not. I would claim there isn't any requirement as of today that those two things are separated and there's nothing in our documentation anywhere that says otherwise. If anybody believes that's not the case, I'd be interested to hear where there is a mention of any kind of requirement. Now, this is not to say that we shouldn't have one but I think it's important to agree on what the status quo is because otherwise, I mean, we have a bigger problem if we don't even agree on where we are. Yeah, so I don't think we define the policy, right? I think why we took it really pushing for a 1-0 because it was a signal to the market, we felt that a few features are mature enough and you know, all the antennas were happy to sign it off. So we said, okay, can you hear me? Can you hear me? Yeah, I can hear all of you guys but Jonathan's coming in really not a very crisp sound. Okay, then I'm not going to say anything. Fine, I don't know what to do. I'm in a loud place, maybe I'll move. I can try to say it like maybe in a short version, I think we wanted to keep it open. Who is you in that sentence? Jonathan. No, I just looked at how we did 1-0. We had an agreement on okay, let's go to 1-0. The beginning we were thinking about March 30th, we communicated like April, so more or less then we added some more features related to the challenge and then everybody said okay, let's get it out of the door. People really wanted 1-0 in fabric, right? But we didn't want to force it. I never had, like even when I was always listening to the TSC I never had the policy kind of either encouraging or discouraging people from going to a 1-0 release. So I feel I believe all the time that we wanted to keep it flexible. That was kind of my assumption. But if you want to define a policy then maybe we should, if people think that we should discuss that because there are pros and cons, right? There are pros and cons to encouraging people to go 1-0 and it's harder to go 1-0. Just to get out of the incubation, right? However we called it that step. What I said last week, I think it is a big step. I think it is a game changing event. It's not like 0.92 is not the same as 1.0.1. If you see what I mean, right? It does the release cycle. As a signal to the market, there's how much money because of auditing and all the other things that come. And the good things that come along with the 1-0 release cycle and getting out of incubation. The question is, do we want to encourage you or do we want to make it this strict process? I don't know how people feel, but I'm open to suggestions. And I'll mute myself now, sorry. Yeah, so I think to the earlier point, we do not have a policy on it but what we're discussing is, do we want to create a policy? And I think it makes sense that if we're going to put some sort of marketing and backing behind things, or just even the notion that hyperledger supports the concept that is particular project is production ready, that it has to have some level of maturity behind it. And so it makes sense to me that a project be in an active state at the point that they're declaring a 1-0. Or the opposite of that is it seems like it would be odd for something to claim that it's production worthy when it's still being incubated. So I'll just sort of go back to the conversations that we had around incubation and maturity of the project itself from a code. And we made a clear distinction that they're unrelated. Exiting from incubation doesn't mean the code is mature. It means that the community is mature. It doesn't necessarily mean the code is production ready. It means that the community has met the criteria that we outlined in the project lifecycle. We didn't talk about, you know, releases in that context, but you could, I think, make an argument to say that code maturity is not a requirement for exiting incubation and then why should exiting incubation be a prerequisite for community or project community maturity? And again, I think that these are independent contexts. Again, I think I can appreciate that we wouldn't necessarily want to be in a situation where somebody comes in and is not putting in, you know, immediately after they incubate the project, they go to 1.0 and expect all the marketing and so forth. I think that's probably something we want to try and avoid. And, you know, I think there needs to be a certain expectation that there's active efforts on going to improve the community maturity around the project. But again, I'm just mindful of the fact that we made a clear distinction between code and community when we made the decision around whether or not the code had to be at, you know, sort of production grade before you could exit incubation. And I guess I'm saying why would we have that a different discussion in the other direction? I think I would just point out that when the project goes to 1.0, that it does send a certain signal to people who are using that product that it will be, that it will exist, right? It continues to exist and be something that can be used, right? And so this is the only reason that I think, Chris, I completely agree with the fact that we've written that incubation and code level kind of don't go hand in hand, right? But I think on the other end, you have to think about the people who are using the project and thinking, oh, this is 1.0, this community is going to continue to exist and I can go ahead and base my product on this and work from here, right? So I think that's the only thing that I would bring up as a possible concern with the other way around, right? Not being active for 1.0. And well, the counterpoint is that a project just could be small, right? Like if I, if, you know, say Mick and I write some small little library that's like 300 lines of code and doesn't need much maintenance or something, but a lot of people use it. Well, you know, maybe we don't have a large number of contributors just because we don't need it. And if we were to go away, then someone else would just pick it up because it was simple. Yeah, I think we shouldn't get too focused on the number of contributors. Diversity is important in ensuring that the project remains if one participant or one company goes away. But I think the other things we look for in an active project are maturity in the processes. So you've got the CIA things covered. You've got sort of, you know, build integration, unit test coverage. Your licenses are checked out. And all these things are, are part of a picture of maturity that seemed to me as precursors for declaring production level maturity. Dan, I completely agree with that, but that if the code is mature, this process is going to go very fast, but there's still a process to be followed and a bunch of gates that we've put in place that, that trigger actions, both in that, you know, both as a sort of reflection of code, but also trigger actions on the working groups part from security audits to marketing pieces of it as well. A 1.0 in an arbitrary release is fine, but a 1.0 coming from Hyperledric contains at least some stamp of blessing that we should be able to address. I would like to suggest that we may have a look at what, what happened in other communities, like the Omistak and Apache. So in my opinion. So I think we lost your audio, Bawa. It looks like you may have gone back on mute. So I may encode in the chat later. Okay. Sorry with my microphone. Yeah. So I don't think we're necessarily ready to completely close this off. I was like, I think there's, you know, there's good points on both sides. Right. I suggest we continue on the mailing list and pick it up again next week and hopefully I will have better audio. Any other discussion topics? If not, I can give people a couple minutes. Right. Thanks. Thanks everyone. Thanks all.


Do you find yourself burdened by an aching sense that something around you, something with someone else is very very wrong? Well it could be that you are experiencing paranoia. Something that I'm sure many of you understand, have knowledge on of experience to some degree in specific social situations, especially when you're up in front of the class delivering a PowerPoint that you put together about five minutes before the lesson. Autistic paranoia, the experience of going through life, having those really early childhood experiences, particularly in secondary school or high school as you Americans like to call it. And you develop a lot of paranoia due to that atmosphere. You have a lot of social dynamics that you don't really understand and you don't really trust people because you don't know what's happening. You don't know all these social rules that are happening. And going through that for such as a prolonged like period of time, undoubtedly it's going to instill a lot of emotional or even physical changes to your brain. A lot of emotional changes that sort of carry on into adulthood. When I was in secondary school, my earliest experiences of paranoia occurred when I became acutely aware of the fact I didn't live on the same wavelength as everybody around me. I didn't get why people were laughing. I didn't get why people were saying stuff in a certain way. I didn't even pick up on it. I didn't understand how people knew stuff about other people that they've never talked about gossiping for you in order to kind of combat this because I was getting myself in situations which I received some public humiliation, online humiliation. In pretty much all circumstances, people really just took advantage of me due to my lower level of social competence. And so I suspected danger everywhere to keep myself safe. The moment that I got on that bus to school, to the moment I came back, I was on high alert all the time and it caused me a lot of stress, anxiety, meltdown, shutdowns, everything like that. But I needed to be aware because I didn't feel safe. I didn't feel safe in my secondary school. I don't think it's something specific about mine because those are a lot of great things about it. But it does seem to be quite a common occurrence that we receive a lot of trauma in secondary school and then doubtedly that's going to come a lot with feelings of paranoia. So I guess before we jump into the meat of it, let's clear out the air. What is paranoia and why does it happen? Paranoia is the feeling that you're being threatened in some way, such as people watching you, acting against you, even though there's no proof that it's true. And this is a difficult one because the social landscape, it's not always certain. And nobody has the power of oversight, but if someone was to walk through life being constantly paranoid about something that could happen, such as being bullied or hit, at some point it's going to happen. It's going to validate those feelings of paranoia. Even though perhaps for the entirety of the school days, it's not really something that happens on a very frequent basis, but you're always in that state ready to combat it, ready to protect yourself as it be. So when we become paranoid, it's a really toxic exercise in trying to read the future or see past the superficial. And when you're autistic, especially at that age, it's going to be very hard for us to identify feelings and identify social dynamics, things of that nature, unless we've been taught it. And even then, it's going to be extremely difficult. If you're paranoid, you're going to focus on those thoughts, those feelings, and it's going to go in a loop and it's going to be a lot more of a constant thing even each day. You may not even have a diagnosis of paranoia because you don't fit the criteria, but you do feel paranoid all the time, but you don't really know why. And the thing is, yes, nobody has that power, so it leads to a constant state of anxiety and mistrust. And that anxiety and mistrust can carry on into adulthood. Going into, I guess, one of the more specifics about autistic paranoia. I've already said it, but your natural instincts for people will be off. We base how we think people will act based on how we think about things and how we perceive things. And if you don't and you're different, you're not really going to have a good sense for it because people aren't like you, the majority of people like 99, 98% of people. Do you have autistic paranoia? Do you hate or suspect pretty much all of the NTs, all of the neurotypicals that you come across?  Do you think that words or affirmations that people give you are like an illusion, like they're just saying it for the sake of it? Do you doubt your own interpretations of things that are happening around you? Even now as an adult and doing all this research in psychology or understanding a lot more about neurotypicals and autism and things like that, do you still doubt that your decisions, that your perceptions of things are always wrong? Do you think that they're always wrong? I'm pretty sure you got the message there. Paranoia is a very difficult thing to overcome, but it is possible. It is possible to shift your mindset. Obviously I'm not a licensed psychotherapist or anything of that nature, so it's best to take advice about mental health conditions like that from someone who knows a bit more about it. But I'm just speaking from personal experience and the experiential knowledge that I've gathered from talking to numerous autistic advocates and people. So to get over it, you really need to develop some sort of trust in people, and that requires a lot of patience and it requires you to trust in yourself as well. We can never be completely certain of things, but we can have a good idea about it and we can think of things that are more likely to happen than other things. The issue when we grow up and we have all these traumatic experiences in childhood and adolescence is that we have a very negative experience of people, and so we're a lot more closed off, we're a lot more, I guess, dismissive of people, particularly neurotypicals. It's a really tough thing. You may even find that when this thing occurs that you're worried about, you feel like the veil is lifted. If you find yourself in the same situation as time and time again with people, then learn from it, but you don't need to be paranoid about things. You shouldn't be overthinking lots of things in your head over and over again. If you have a query, talk to a counselor about it, talk to a friend about it, talk to an autism organization, talk to someone about it, or at least just get it written down on paper and sort of try and dissect it from there. The issue is with anxiety and paranoia, things go around in your head over and over again, and if you don't have any output or input, it just kind of stays like that. It's very common for us to be quite mistrustful of neurotypicals, because for the most of it, during adolescence, we receive a lot of negativity from them. I love sometimes a lot of trauma and sometimes abusive situations. The real step for this is to go out there and try and find positive experiences. It may be hard and you may be quite cynical, not very positive about the possibility of connecting with another person, but you can find those good experiences and you can find those good people. It's just about trying to learn about how to spot the red flags in people, how to do all of that, and how to retreat from a situation if you need to. Obviously, this is quite an exploratory topic. I'm sure that there will be something out there and perhaps a research study that I haven't read yet. I do think that it's worthwhile considering whether you have a negative bias towards a particular group of people and why. Some of the most amazing people that I know are not autistic, they're neurotypicals, and some of the other amazing people that I know are autistic.  There's a lot of elements to this and I think it's just worthwhile highlighting situations where things that you've learned in adolescence are being transferred to adult life and causing you to miss out on a lot of opportunities for friendship and connection and just generally feeling at one and happy in today's society. So yes, very exploratory, not very concise and I feel like my brain's going all over the place for this video, so I'm going to end it here and please go follow my social medias. This is based on a post that is probably a lot more coherent and makes a lot more sense than the video that I'm doing now. It's basically kind of a loose, these Instagram posts are basically a loose script for my videos. So go over there, it's mostly Instagram that I post on, but you can follow me on other places. It's all at Thomas Hanley UK and if you want to get in contact to be on the podcast or to get me for modelling or public speaking or to attend certain online events, go through my website, thomashanley.co.uk. There is a contact form on there and yes, I mentioned about a podcast, 40 Autie Podcast Season 2. It's out, I've had some very notable guests on there, some big names around autism and also love great conversations with a lot of different autism advocates from across the globe. So go check that out, all of these links will be down in the description and I hope you have enjoyed this topic and please comment down below letting me know what are your experiences with paranoia. Do you think that these feelings towards narratives are warranted or do you think maybe it's time to sort of try and get rid of that negative bias? That's a very loaded question, apologies for that. Anyway, I'll see you later guys, peace out.


All right, hello and welcome to administering multi-cluster service meshes securely. I'm Aitan Yarmush, an architect at Solo IO. I'm Eric Murphy. I'm a field engineer also at Solo IO. So before we jump in, let's just quickly go over what we're going to discuss in this talk. First, we're going to review multi-cluster service mesh and how it has evolved. First, focusing on single-cluster service mesh and jumping off from there. Next, we're going to discuss the limitations and security issues with the current approach. Third, we're going to explain how we at Solo IO solved those problems. And then fourth, we're going to give an example or demo of how to deploy securely using GitOps. So before we jump into multi-cluster service mesh, let's just quickly take a look back at single-cluster service mesh. So this is a very simple single-cluster service mesh example. We have a couple of different workloads running in cluster as well as our service mesh. In this case, the service mesh is Istio, but that is not necessary for this. So as you can see, all of the workloads including the Istio control plane are communicating directly with the Kubernetes API server. In the case of the control plane, it communicates with the API server to get the endpoint information for all the workloads as well as service info and other metadata about the cluster. So let's just quickly review the single-cluster scenario and what the benefits are and why it's more simplistic. So the control plane and the data plane both live on the same cluster. Authentication with the API server is handled automatically by Kubernetes. So the service account tokens are generated and mounted automatically. And the tokens remain safely in Kubernetes, more on this later. Most likely no need to revoke those. And lastly, configuration can be easily housed in a single repository, leading to simple, easy to understand GitOps workflows. So now that we've quickly discussed a single-cluster Istio scenario, let's evolve, let's jump that up to a multi-cluster scenario and begin to explain the complexities and security implications involved. So now, as you can see, we've taken our single-cluster diagram and multiplied it by five. Now, in this scenario, all of the control planes still need access to the cluster that they are running on. But in order for a multi-cluster service mesh to be useful, the Istio control planes also need to be able to read and write to the other clusters. So what does that look like for us? Well, we've quickly drawn up an example. So now we can see all of the Istio control planes have access to every other cluster's Kubernetes API server. So what does this mean in practice? Well, in practice, this means that every control plane needs to have credentials to access those other clusters. So, as we mentioned earlier, the Kubernetes credentials need to leave the Kubernetes, the single-cluster boundary. So multi-cluster service mesh. The control plane and the data plane are spread out across many clusters. Each control plane needs access to each other cluster. That means we have an order n-squared connections where n is the number of clusters. Authentication is not handled automatically for our remote control planes. So service accounts must be created. Kube configs containing the service account token must be distributed to each cluster. Now, these Kube configs, which must leave cluster borders, give potentially dangerous levels of access to said cluster. If any Kube config credential were to leak, replacing it and redistributing the credentials is a convoluted process. Keeping all of these Kube credentials in sync across all the different clusters is very difficult and not easily done using GitOps. And on top of this, in deployments with many clusters, potentially many control planes are performing IO on each API server, which can slow it down. So the approach that I just mentioned with order n-squared connections, that's one way of doing it. That was the Istio way of doing it beforehand, but a new way is starting to emerge, which we think is a lot better. And that's the external control point. So in this scenario, Istio is running or the service mesh is running on a management cluster and managing a bunch of clusters on its own. And in this way, there doesn't need to be a control plane on each cluster that needs access to each and every other cluster. So there are, of course, downsides to this scenario, such as if the management cluster were to go down, then the control plane would also go down. So it is important to build in redundancy for the management cluster, but you also reduce the number of places that need the CUBE credentials, as well as reducing the load on the API server. So the external control plane, the control planes are now on a single cluster and the data planes are still spread out. Now, we like to call this the push model of configuration. There's one centralized control plane with access to all data plane clusters, so that's order n connections. And the centralized control plane will be pushing config out to the remote clusters. Each control plane will still need access to the API server of each node or remote cluster. And although the number of network edges has shrunk, the same problems from the previous model still exist, albeit on a smaller scale. Still hard to distribute the credentials and the leaked credentials can give bad actors outsized power. So what are our concerns, our main concerns about administering multi-cluster SDO or service mesh securely? One, difficult to store and sync all of the CUBE credentials using traditional GitOps workflows. And two, giving a cluster read-write access to many other clusters leaves a potentially large security threat. So how can we get around some of these issues? Well, the way that we thought about it is something called a pull-based method of config. So if you notice in the previous diagram, all of the arrows were pointing out to the remote clusters, whereas now all of the arrows are pointing in at the management cluster. So it's still an external control plane, but it's pull-based this time. So now all of the remote clusters are actually creating the connections to the management plane. So the management plane doesn't need to have access to any of the remote clusters and neither does the service mesh running there. So what benefits does this bring us? Well, this pull model of configuration, as I mentioned, there's one centralized control plane which needs access to zero data plane clusters. All the data plane clusters connect to the control plane cluster and receive updates that way. The config updates function similarly to the Envoy XDS protocol, for those unfamiliar, we'll go over that in a minute. And linked credentials no longer give the same direct control over the API server. Furthermore, no Kubernetes credentials need to be shared outside of the cluster borders. And since we are no longer distributing these credentials, the remote components we are distributing can be easily stored and distributed via GitOps. So how does this really work? Well, we took heavy inspiration from Envoy's XDS protocol and the Kubernetes Bootstrap token, which I will explain now, but those links are available from the slide. The way that the XDS protocol works is essentially Envoy makes a connection to the control plane or the management server and says, hey, I want some config updates. And then the management server says, all right, great, I've got that for you. And so in that way, each remote instance, each Envoy is responsible for creating the connection. And then the management server is responsible for updating Envoy with the config when necessary. And the Kubernetes Bootstrap token is a way of verifying the identity of different callers. So more on that in a second. So how does it really work? Well, in this case, the server would live in is our management plane. That is the component living in the management cluster and the agent. And the agent is the component that gets deployed to each remote cluster or each data plane cluster. So we start off with a CA or signing cert on the server and a server cert on both the agent and the server. And the server cert will be used to establish the initial TLS connection. In addition, we distribute a Bootstrap token to both clusters. And this Bootstrap token will be used in the initial request with the token so that the server knows to trust the agent. And then once it knows it can trust the agent, it will respond with its M TLS cert and it's with the identity. And this M TLS cert is how the server knows what data to send back to the agent. So once that identity has been provided, then we can open our config streams. So the first one we have is a remote resource stream. And this will stream the resources from the data plane clusters from the agent to the server so that it can make decisions about the system. And then the second one will stream the mesh config or the config needed to run the data planes to the agent clusters. So can the pull model really solve all of our problems? Short answer, no. But it does alleviate the two main issues that we began to discuss a few slides back. The first being giving a cluster read write access to many other clusters leaves a potentially large security threat. Well, since our control planes no longer need direct access to remote API servers, the Kubernetes credentials no longer need to leave cluster boundaries. This reduces the chance that a bad actor gets elevated access to the system and reduces the load on the API server by reducing the number of clients. The second, difficult to store and sync all these Kube credentials using Github's workflows. Well, luckily, there's no longer any need to store or sync these Kube credentials on top of which the components that do need to be synced, mainly the agent and the token, can easily be integrated with Github's C part two of the talk. So what were those question marks that we were talking about earlier? Well, it just so happens that we at Solo IO have created a product called Gloomesh, which can handle this, but more on that later. For now, I'm going to kick it over to Eric. Thank you, A-Tun. So next, I want to talk about how you can deploy Gloomesh securely across multiple clusters using GitOps and Argo CD. So I have a demo I want to show you today. And in that demo, I have two clusters running. I have a management cluster. And on that management cluster, I have Argo CD deployed. I'm going to use Argo CD to deploy configuration from a Git repository and automate the deployment of the Gloomesh management plane. On the remote cluster, I have a separate instance of Argo CD running. And I'm going to use configuration there to deploy the agent automatically. So there are multiple benefits here to this approach where you can take configuration as code, you can store everything in your Git repository. And you can have a repeatable deployment process for both the Gloomesh management plane and the agent. And this allows you to automatically update your Gloomesh deployments with that configuration with Argo CD. Additionally, if you need to spin up new clusters and deploy new meshes, you can do it in an automated way more quickly and more effectively. Additionally, in a disaster recovery scenario, you want to be able to replace your clusters and your meshes as quickly as possible. And you can do that with this approach, even including that management cluster. Okay, so now let's jump into the demo. So what I'm going to show here in this demo is actually how to physically deploy Gloomesh using Argo CD. And so to do that, I have some configurations checked into my Git repository. Everything is deployed from a Git repository using Argo CD. And in that Git repository, I have CRDs that are specific to Argo CD. And so this is an application CRD. And inside of that application CRD, you can actually define values for a helm chart. So behind the scenes, Argo CD is actually using helm to deploy the Gloomesh management plane and the agent. So in this case, I'm using the Gloomesh enterprise chart, right. And I'm setting a license key and some other values such as self sign equals false. That means I'm providing the certificates in advance of the installation. I'm not depending on Gloomesh to generate certificates on its own. Okay. And then if I go over to this other application CRD, I can see the helm chart for deploying the agent. And so here the chart is enterprise agent. And I'm giving it a server address for the management cluster. I'm giving it an authority for the certificates that are used. And I'm giving it a cluster name, which is simply remote cluster. Okay. So now let me walk you through the steps for actually conducting the deployment with Argo CD. Okay, so first, I'm going to switch my kubectl context over to the management cluster. Next, I'm going to create a new name space for Gloomesh on that management cluster. Next, I'm actually going to kick off the deployment of the Gloomesh management plane using Argo CD. And I can do that by simply doing a kubectl apply on that application CRD. Okay, so that's done. So what I can do is go over here and log in to the Argo CD deployment on my management cluster. And you can see how quickly that was actually deployed. Everything is synced and healthy already in a matter of seconds. And if I click in here, I can see everything that was deployed for the Gloomesh management plane. Okay, so you can see there's a lot of different things deployed here, right? Everything is green. Everything is deployed. Everything is healthy, including all the pods for the dashboard, the enterprise networking. And so that all looks good.  And so there are a lot of capabilities built into it. Then if I step out here and look at the rest of my application deployment, I can see configurations for the Gloomesh management plane. So here I can see that there is a RIT certificate that is stored in a secret. I can see that there is a TLS certificate for securing connection between the remote cluster and the management cluster. There is a signing certificate also stored in the secret. And that allows the Gloomesh management plane to generate new certificates that can be passed to remote clusters to establish an MTLS secure connection. Okay. Also, there is a token that is created. And I provided this token in advance. So that token is deployed with the Gloomesh agent. And that is used for making that initial connection from the remote cluster to the management cluster, as that was discussed previously. Finally, I have a configuration called Kubernetes cluster. And that simply defines the remote cluster that is configured to be registered by the agent. And I'll walk through how that works. Okay.  And so that will allow me to see what is currently running, what is currently registered in the Gloomesh management plane. Okay. So as I mentioned before, there is a Kubernetes cluster CRD. You can see that the cluster is there, right? But if I go in here, there are no meshes that were discovered, right? And that is because the agent has not yet completed registration with the Gloomesh management plane. Okay. And so that is going to be the next part of the demo. Okay. So let me go back over here. And so what I'm going to do is I'm going to change my cube cuddle context to be for the remote cluster. I'm going to also create a Gloomesh namespace there. And I'm going to deploy that other application CRD specific to the Gloomesh agent. Okay. So let me paste that in there. There we go. All right. So now let me go over and log into the other instance of Argo CD. This is not the same Argo CD. This is the one deployed on the remote cluster. Okay. So I'm going to click sign in here. And here you can see that everything that is needed for the Gloomesh agent is deployed, right? So that also happened very quickly in a matter of seconds. You can see that there's enterprise agent pod that's running right here. So that is green, that is healthy. So everything looks good for the Gloomesh agent deployment. Then if I go over here and look at the rest of my deployment, I can see the actual configurations that are residing in that repository deployed by Gloomesh. Okay. So if you look here, there is a secret that is deployed with that RIT certificate, right? And there's also that identity token secret. So that is the same token that was deployed on the management cluster. And once again, that token is used to authenticate the remote cluster with the management cluster and establish that initial connection with the management plane for Gloomesh. Okay. And when that happens, it receives back an MTLS certificate and a new connection is established. So there is a fully secure connection for the Gloomesh agent. Okay. And one thing I want to call out here, just for demo purposes, I'm putting in the certificates end-to-get for the RIT certificate. That is not the best practice. Instead, you could use remote secrets. You could integrate with Hatchecourt vault. You could integrate with your cloud provider with their certificate management services. So once again, that's just for demo purposes. Okay. And by the way, all this is stored in the get repository as I mentioned. So if you go in here, you can look and you can see these configurations for the Gloomesh agent. You can see like the token, the token value right here, right? You can see the root TLS secret. You can see the CA cert right here, right? So all that is deployed using that that get ops approach. Okay. Okay. So since we deployed that agent, it should be registered. So let's go back and look at the Gloomesh console again. And you can see now that under meshes right here, I actually have a mesh that's registered, right? It's this IstioD IstioSystem remote cluster right here, right? And the mesh health is green. And I can see that there are some destinations that were automatically discovered on that remote cluster. Keep in mind, this Gloomesh console is running on the management cluster. So if I go in here and look at the mesh details, I can see that there is a destination for the pet store application. I deployed that pet store application in advance of this demo. It is part of that service mesh offered by Istio on that cluster. The sidecars were automatically ingest injected into the pods. And so that was how the pet store was automatically discovered by Gloomesh. Okay. So that wraps up the demo. So let me go back to the slides and just finish up here. So Gloomesh is a really awesome product for providing that multi cluster, multi mesh management, right? But Solo offers other products that integrate with Kubernetes, integrate with service mesh. So we actually have an API gateway product called glue edge that offers integrations with Istio. We also have something called the glue portal. So that provides a developer portal, a user interface, an API specification discovery that integrates with glue edge and also Istio ingress. And finally we offer something called web assembly hub. So that is a place where you can build web assembly extensions or the onboarding proxy, you can publish them, you can deploy them. It's compatible both with Istio and glue edge. So with that, I thank you for watching today. And if you have any questions, please feel free to reach out. Thank you.