So the topic for today is how to disclose or sell an exploit without getting in trouble. I'm Jim Dinaro. I'm an intellectual property attorney based out of Washington, D.C. I focus my work exclusively on information security technologies. Before I went to law school, I used to spend far too much time just tweaking around on MaxBug, on my power PC and figured that no better way to keep doing that than to do this. So here we go. Just because I'm an attorney and this does have some legal component to it, although this is not a law talk, really, I have to give the standard disclaimer that this presentation is not legal advice about your specific situation or your specific questions. Even if you ask me a question, we're still talking about hypotheticals. If we develop an attorney-client relationship, then we're talking about your specific problem and giving specific legal advice. So this presentation does not create an attorney-client relationship alone. We can maybe do that later. So this is a quick overview of what we're trying to accomplish here in the next 20 minutes. I'm going to speak quickly to make sure we get it all in. We're going to cover the types of risks that are being faced by researchers, some risk mitigation strategies that researchers can take to try to reduce those risks, some of your options for disclosing your vulnerability that may have less risk, and then some of the risks that are associated with selling an exploit. The overall goal of this is to make yourself a harder target. If someone ever asks you, well, can I be sued if I do this or if this happens? The answer is always yes. You can always be sued by anybody for anything at any time. The only question is who's going to win? And the goal is to make it more likely that you will win, which disincentivizes someone from actually suing you in the first place. So let's start out with just some great examples of the kind of research activities that might get somebody in trouble. For example, some of these are generally real-life cases. You found out how to see other people's utility bills by changing the HTTP query string. I talked to someone at a party the other night who had done just exactly that. He was wondering what to do about it. You discover your neighbor's Wi-Fi is not protected. How did you find that out? You broke the crypto that's protecting some media that you had. It's getting a little more serious now. There's actual money at stake. And maybe you wrote a better remote access tool that sounds like you might make a lot of money. So many of the same risks apply, surprisingly enough, whether you're just looking at changing HTTP strings or whether you're actually taking apart a DVD. So in general, we're talking about techniques. I've sort of defined it here. Broad spectrum, everything from denial of service, a technique that might be used for a denial of service attack to something that's just, you know, more akin to sort of investigatory web browsing. Okay, first, when is there risk to a security researcher? There are three general areas where we see the risk starting to show up.  There are some examples listed here. You might be the recipient of a legal action seeking an injunction barring you from disclosing something before a conference. So we now remove from merely saber rattling to an actual lawsuit being filed against you. And then there's the possibility of a legal action being initiated against you after you make the disclosure. And these are all real examples. You can, Declan McCulloch of CNET and his colleagues have written some very interesting articles that go into more detail about some of these cases. I would recommend them to you most definitely. But you might notice that some of these seem to be around black cat and Declan on a pretty regular basis. So that's when it can happen.  You've probably heard a lot about that lately, perhaps here at other conferences. The main issue is that it prohibits access without authorization or exceeding authorized access. The two times when you're likely to run into possibly exceeding authorized access or acting without authorization would be in the investigatory phase of working on your whatever technique it is that you've got. And when you actually create a tool that performs whatever this technique is, you might actually have a problem where that tool does the act that is prohibited. So in light of, you know, everyone's talked much about how vague this notion of the computer fraud and abuse act is of authorization. I've created a handy checklist to figure out if you might have a computer fraud and abuse act problem. So there we go. Are you connected to the Internet? Probably. Are you accessing a remote system? Probably. Do you have permission to access that system? This is a real hard question. It's really hard to know if you have permission. If you saw a banner go by that said you don't have access, you probably don't have access. But there are a lot of cases where it's not so clear. And that's really where you have sort of like the Andrew Arnheimer situation where he's querying a public facing API on a repeated basis. No one asked him to do that. But there was no banner. There was no clear prohibition of doing that. It was a public facing API after all. So it's really, there is some real risk in figuring out if you can, whether or not you have permission. But that's really all it takes. Unfortunately, it's not just about what you do. The computer fraud and abuse act is about what your friends do. And I believe the risk of being caught up in conspiracy to violate the computer fraud and abuse act is most certainly enhanced by the prevalence of social media today. So if you're on Twitter or some other easy, very easy to use social media platform, you're talking to your friends about how you might do something or answering questions about how you might do a certain thing with a technique that you've developed.  Conspiracy typically does require an overt act in order to really fulfill the conspiracy and typically just discussing something with someone is not. But if you start providing technical support for something that someone else is doing, you're really definitely increasing the risk of being caught up in a conspiracy to violate the computer fraud and abuse act if not actually violating yourself. So we've got some examples here where computer fraud and abuse act has been applied. I think it's helpful to look at some examples because that's how we see how it's being applied. And we can compare what we're doing to some of the things that have happened in the past, other people and see how close those comparisons are. And since we're in Las Vegas, we absolutely have to talk about the case of Nester. Nester was really into video poker. And he liked to play and play and play and play. And he got really good at it. And he played it so much that he discovered a bug in the video poker software that enabled him to play one type of game and bid up a bunch, bid a bunch of money into that game and then switch to a different game and a multiplier would be applied to his bid. So when he won, he got to see enormous payout. And he figured out how to reproduce this bug very efficiently. So he was doing it and his friends were doing it and they were getting a lot of money. And obviously eventually, how these stories always end, he gets caught. And he was charged with, amongst other frauds, violating the computer fraud and abuse act. And as we were just looking at the computer fraud and abuse act a few moments ago, we saw it's really mostly about unauthorized access or exceeding an authorization that you had. And it's hard to imagine how sitting, he didn't access the firmware, he didn't take the game apart, he just sat there putting money in and pushing the buttons on the surface of the machine. How you could exceed authorized access to a video poker machine is absolutely mind boggling. But nonetheless, those charges were levied against him. Ultimately, the Department of Justice did not pursue those charges. Those charges were dropped. They went ahead with other fraud charges. But nonetheless, for some period of time, he was facing computer fraud and abuse act charges for doing exactly that. It's also worth looking at the tragic case of Aaron Swartz, who spoofed his MAC address to download journal articles.  Andrew Orenheimer, who allegedly conspired to run an automated script to plug in identifiers for iPads and get email addresses, didn't even do it himself. He's doing several years in federal prison for that.  There's really no end to the sorts of things that could possibly violate the computer fraud and abuse act. So you're looking at a situation where the computer fraud and abuse act almost acts as an ex post facto law where the Department of Justice is able to look at what you did after the fact. And if they don't like it or they don't like you for whatever reason, you may be trollish for some reason.  There's also a civil cause of action provided by the computer fraud and abuse act. So the company that, wherever the target is of this exploit can also pursue whoever accessed the system without authorization. So the question then is, what can we, is there anything we can do to try to reduce our chances of being on the wrong end of this type of lawsuit? Well, let's take a quick, we don't want to go too far into statute. It's not a continuing legal education conference, but let's just take a quick look at the statute and see if there are some key words we can at least identify. Here we have whoever having knowingly access to computer without authorization. Another part of the statute, whoever intentionally accesses a computer without authorization. So one of the things you can do is to try to avoid unintentionally creating knowledge and intent. It's a little bit hard to do this for yourself if you intend to do something, but at least you can avoid doing it in connection with other people. So, for example, I would suggest that you do not direct information about how to use some kind of technique to someone that you suspect or have reason to know is likely to use it illegally. Be careful in providing technical support for some clever new technique that you've developed. So my advice, if I were your lawyer, I would advise you not to answer that tweet. If someone's tweeting to you asking you about how to make something more effective, perhaps. This slide's a little more detailed. Some more approaches that you might take. Don't provide information possibly directly to individuals, especially if you're not sure who they are or what they might be up to. Consider just posting things on a website only. Do not post information to forums where you suspect or forums that are known to generally promote illegal activity. If you publish it on your own website or you have control of the post, consider disabling comments so you don't have a situation of people discussing potentially illegal uses of your technique. And lastly, don't maintain logs. So one of the things we've seen happen, so that's enough for the computer fraud and abuse act for now, there's not a whole lot you can really do about it beyond just being careful. Let's move on to the temporary restraining order. This is particularly timely, actually, because you may have read the story about the BW group and the megamos encryption that was used on the vehicle immobilizers. So some European security researchers had figured out how to bypass or discover a flaw in the encryption that was used on the vehicle immobilizers that were used in BW group cars like Porsche and Audi and Bentley, and they're going to present this at the USENIX conference in Washington D.C. in a few weeks, and they got themselves slapped with a temporary restraining order preventing them from making this disclosure at the conference. How did this happen and how can we prevent this from happening again?  So take a quick look at the factors that the courts look at when deciding whether or not to grant a temporary restraining order to prevent a researcher from disclosing some information about a vulnerability. Number one, will the requester, so in that case it would be the BW group, suffer irreparable harm if the TRO does not issue? Who knows how this works? Is he a new speaker? It's really hard to get accepted to speak at DEF CON, right? You guys should all be thinking about how you're going to create speech, talks. I've been drinking all day. Talks to eventually have yourselves up here, right? A big round of applause for Jim. One more order of business. We need a new person who's first time at DEF CON. First hand up right there. Red shirt, come on up on the stage. We've got a little extra. Let's get one more. We're going to get two people. All right, first hand up over there. There we go. All right, cheers to our new speaker. Let's see if you can pick up where you left off. We're going to work on new material for tomorrow. Thank you. Thanks, guys. All right, so that was great. Thank you. So just a quick look at some of the factors that the court's going to look at when deciding whether or not to grant the temporary restraining order. There's someone like the BW group who wants to stop a presentation for happening at USENIX. Will the requester, the BW group suffer irreparable harm if the TRO does not work? Pretty easy to imagine. You've got an embedded system and someone's figured out how to break it. It's going to be almost impossible for them to update it in any reasonable amount of time. They're usually expensive. Probably some irreparable harm. That basically means that money isn't going to fix it very easily. So that goes in the BW group's favor. Will there be an even greater harm to the researcher if the TRO does not issue what your paper got delayed? You couldn't put in some part of the paper. Hard to see that as a huge harm to the researcher. We might feel really bad about that. But in terms of the use sums of money, BW group's going to have to pay to fix this. It's not really going to look too good for the researcher there. The public interest is just kind of a fun one because we might think that, well, the public interest clearly favors disclosing the vulnerability so it can be fixed. There's, of course, probably going to go the other way on that and see that really risk to all these BMWs or other Porsches and Bentley's and things being stolen is much greater. It is much more in the public interest than having your really obscure crypto talk go forward. The last factor is the likelihood they request a role to ultimately prevail. And this is really the one we need to focus on because the BW group has to have a cause of action. They can't just say we don't like it. They have to say, here's why you need to stop. And it's because you did something bad to us. And in the case of the BW group that would make the most case, and also in the case of the Cisco disclosure, what we had was the use of copyrighted material. And that's really what, that was the hook that got the TRO to issue. So the obvious advice then is to avoid the use of copyrighted material. So if you include source code or object code from whatever it is that you're working on, that gives leverage to whoever it is that wants to stop you from disclosing it. There is a fair use exception if you use little bits and pieces of code. That's a case-by-case analysis and you can't just say, well, this is going to be fair use. It depends on how much you use and other factors that are very specific to what's actually going on in your case. So just try to avoid it if you can. It may not be possible, but to the extent you can do that. Also avoid darknet sources for where you're getting this stuff. In the Megamos case, the court actually talked about the fact that the researchers obtained some information about how the Megamos system worked through some sketchy channels. I don't know how to call it saying exactly where they got it, but it was some sort of BitTorrent P2P type thing where they got it. It wasn't from 3W Group or Megamos. So another thing you want to do is be aware of pre-existing contractual relationships that you as a security researcher might have with the target of whatever it is you're working on. These contractual agreements could come in the form of a term of service and user license agreements, nondisclosure agreements, or employment agreements. What's that? Sure, so an in-user license agreement might very well have provisions in it that prohibit reverse engineering software, for example, and that's something that you might very well be doing as part of your exploration into your technique, and that could give leverage to someone to try to stop using, oh, you've breached this. You know, nothing's for certain. It's just an argument that they have. I mean, pretty much every piece of software you get is going to have some kind of license agreement that's going to, assuming you came to it legitimately, right? You've agreed to this license that may prohibit you from doing certain things with that software, and there's not a whole lot you can do about that, but you at least can be aware of the risk if nothing else. So how far you need to go in trying to mitigate the risk somewhat depends on the techniques that you've used in your research. If you've done things that are clearly look like some of the examples of what people have done that's gotten them prison time, that's something you need to be careful of and maybe take more aggressive mitigation techniques in order to perhaps hide some of the information about what you're doing. So for example, if in the Megamos case, no one had identified that it was the VW group that was where this crypto system had been compromised, VW group would not have been able to issue to go after a temporary restraining order against the researchers. So perhaps there's an opportunity here for the conference going community to create a track where people could present things that are sort of get like a little asterisk or something next to it and we all recognize that this is something that had to be kept quiet. It's sort of like a confidential disclosure trust the review board, this is going to be really cool but we just can't really tell you about what it is because then you won't get to hear it. So maybe that's one approach. So I'd like to talk about some of the ways that you might make a disclosure that are relatively less likely to get you in trouble. You can obviously disclose a responsible party. That's what we'd like to do. That's sort of what the responsible disclosure paradigm is all about. You found a problem with the system. You tell who's running the system. This is actually unfortunately relatively high risk and that risk scales with the questionableness of whatever technique it was that you used to find out about this vulnerability. So if you're connected to the internet and you access a remote system and you had net permission that's how you did it. It may not be a great idea to go tell them about it because if they don't like it they've got an action against you. If you're inconvenient, that's a problem for you. You might think you're doing them a favor. They might not agree that you're doing them a favor. If you're able to submit it anonymously to whoever the vendor is or the responsible party, that's great. Depends how good your OPSEC is, I suppose.  So that's a risk in itself that you need to consider. If you submit to a bug bounty, presumably they've invited it. Maybe you're at less risk. You can disclose to a government authority perhaps. Maybe you don't really believe it. It'll never get to the vendor. But again, if your techniques were perhaps questionable you might not necessarily want to be submitting it to a government. A governmental authority. You may have an interest in keeping your own identity anonymous. Again, you can try to submit anonymously to the government. But I don't know how much we can really trust that anymore. Fortunately, this is a legal talk and you can almost never get to a legal talk where someone will actually tell you something for sure. Like, absolutely, 100%, you will not get in trouble if you do this. Fortunately, we are in a case here where there is one group of people who really don't have to worry about getting in trouble with the computer front abuse act when they disclose the vulnerability and here they are.  Although she really should not have been hacking the palace computer. We're not going to hold that against her. So we're thinking about ways that we might be able to leverage opportunities for security researchers to make disclosures while keeping the risk as low as possible. So we're working on creating a pilot program where attorney client privilege can be leveraged to hide the identity and the techniques used by a security researcher in making a disclosure. So the concept works like this. The researcher would disclose the vulnerability to a trusted third party which would be an attorney only to the attorney. It's critical that this be a completely confidential disclosure to maintain that confidentiality of that disclosure so that other entities on the outside can't get to it. The trusted third party does not publish the vulnerability on behalf of the researcher. However, the trusted third party does not publish the vulnerability to whoever the affected party is whoever has this vulnerability. The researcher remains anonymous during the entire process. This is possibly of use if there's no better option. It's a little bit cumbersome and there are some side effects chiefly that the researcher remains anonymous. Doesn't get public credit for whatever the research was. It's a possible way for the researcher to be able to disclose and remain about as anonymous as one can possibly get. So this is a pilot program. We're currently working on it. We're kicking out the bugs right now. So if anyone's interested in talking to us further about this we definitely welcome your input and please see me afterwards. We should now turn to selling very quickly. Right now, there is no law in the U.S. that prohibits the selling of an exploit. And that is a situation that is probably likely to change in the not too distant future, but for now there's really not too much to worry about unless the techniques, of course, going back a few slides, if your techniques in developing your exploit have some problem then you still have a problem. But the fact of the sale itself is not something that's going to get you in trouble. However, we've got a focus on this market now and here are some recent articles from May of 2013, booming zero day trade as Washington experts worried. And my favorite, the U.S. Senate wants to control malware like it's a missile. Stuff is dangerous. So every year Congress has to pass the National Defense Authorization Act that sets the budget for DOD stuff that gets stuck in there. And this year for 2014 the Senate version, it hasn't been passed yet it's still in Congress, the Senate version has provisions that seek to begin the process of regulating the sale of exploits. The bill, the House version doesn't have this, this is still just in the Senate, but I think this is where it's headed. The bill notes that the president shall establish a process for developing policy to control the proliferation of cyber weapons through a whole series of possible actions, right, export controls, law enforcement, financial, diplomatic engagement, and so on. The Senate Armed Services Committee that had the bill before was passed to the to the rest of the Senate said they had some commentary on this and they referred to the dangerous software, a global black market, a gray market it starts to look really bad but they note that there is we need to have a carve out for dual use software and pen testing tools. In Europe the European Parliament recently passed a directive they're a little bit ahead of us this prohibition on the sale of tools that the college basically exploits will be required to be enacted by all of the member states in short order and this provision prohibits the production sale, procurement for use import, distribution of these tools that could be used to commit these enumerated defenses which is pretty much all the bad things that you can think of doing with a computer. However, there's an exception for tools that are created for legitimate purposes such as to test the reliability of systems and it further notes that there needs to be in order to violate this law you need to show a direct intent that the tools be used to commit to some of the offenses. So in both cases both in the US and in Europe we're seeing this trend well it's really going back to the definitional problem how do we define what an exploit is and how do we make sure that legitimate tools can still be bought and sold. So this is kind of perspective we don't know what the laws are actually going to look like but I would start thinking like this. Think about dual use tools. If you write something don't put it together as the next greatest hack. You're creating pen testing tools. This has gone on for a long time. If you look at software I'm sure copy 2 plus or locksmith backup software and the manuals for these softwares have these very elaborate disclaimers how this is strictly being used to backup your floppy. This is not being used to make illegal copies and that is really the conundrum and I think that's where software, that's where exploits will go and some exploits will never be able to be a dual use tool for sure. If you have a nuclear missile equivalent of an exploit it's hard to justify the pen testing value of that but for a lot of tools they will fall into this area and that's where perhaps they should go. Some other things you might do if you are selling know your buyer to the extent you can. I think regulation is just one bad outcome away. What's going to happen is someone in the US is going to sell an exploit, it's going to go through some channel and it's going to come back and get used against some US interest. We may not hear about it. It may be a matter of a secret but this will happen and then there will be a huge drive to stop this from happening very quickly. It's the same reason as soon as if someone is murdered with a certain type of weapon that weapon has to be banned that's going to happen here. In this country it's very reactionary and I expect that trend to continue here so maybe you can prevent that from happening so know your buyer if you're selling something don't sell it to a channel where it's likely to go to some country that's under an embargo with the United States. Maybe your best bet is just to sell it to the US ask for assurances from your buyer so you don't have knowledge that is going to someplace where it's not supposed to go to be lied to but you can't control everything, right? But at least you can get an assurance that it's not going to be used in some illegitimate way and also you can always use disclaimer language so I have some nice examples of disclaimer language here this huge chunk of text on the top is actually from a software product that many of you have probably used many times it's good stuff I've highlighted probably the best of the operative language in it but if you're selling something be sure to use some disclaimer language about that kind of flows along these lines that would help you from being charged and being complicit in any sort of illegal use to which the software might eventually be put and lastly I'd just like to highlight this bottom little paragraph which is actually from the Apple iTunes Store it's the end user license agreement that comes with that and it requires that you agree that you will not use these products for any purpose prohibited by the United States law including without limitation the development design manufacturer production of nuclear missiles or chemical or biological weapons thank god words with friends that is dangerous stuff so thank you for coming this is my contact clock I think we have some time here for questions so if people want to line up I'm happy to entertain them as best we can there are definitely free speech issues especially in the temporary restraining order context oh second amendment sorry second amendment challenge come see me after about that question back here what about using a corporation to limit your liability for disclosure or selling has that been utilized corporations can be held liable in many cases in fact even under the computer fraud and abuse act it hasn't happened yet but a corporation could be held liable question regarding full disclosure versus responsible disclosure so when we do it we do it via responsible disclosure we contact the vendor we give them 30 days and we tell them our intent to publish and we publish everything so the actual vulnerability and how to do it to people to replicate and do whatever they want in most cases the vendors get a hot fix within a week and then if within 30 days they provide the hot fix then we print them and say to fix it install hot fix whatever sometimes vendors will say we need more time maybe we'll negotiate a couple of days but sometimes they'll say we're not going to fix it and you can't publish it so I won't explain what we do for that but google recently published the fact that they plan to disclose vulnerabilities within 7 days to have a 7 day turnaround so what happens if a company like google I don't want to use the word threatened but intends to publish a vulnerability within the 7 day turnaround period and that company says to google don't if you do we'll sue you what happens in google versus that vendor company well google is at risk if google has some kind of obligation not to it would depend on the specific circumstances of it but in this case if no law has been broken then google could be published without any discussion in the case for me for example where I contact a vendor and I say I've got the following 10 vulnerabilities which I plan to publish and they come back to me and say if you publish those we'll sue you same thing happens when google say well we're not going to give you 30 days we're going to give you 7 days and the company comes back and says google we're going to sue you if you publish it doesn't carry the same weight where they're trying to sue google as they're trying to sue me for example that helps as well man that's the unfortunate part but is it just the case of how good your legal team is exactly how much do you charge where do you want to meet them at oh let's go wherever you want to meet them at to carry on so this talks over with we've got to get it ready for the evening now in the hallway we'll answer the rest of your questions unfortunately there's not a Q&A because it's been disassembled too thank you


Live from Las Vegas. It's theCUBE, covering VMworld 2017. Brought to you by VMware and its ecosystem partners. Hey, welcome back, everyone. Live here in Las Vegas for VMworld 2017. I'm John Furrier of theCUBE. Live coverage of VMworld 2017 with my co-host, Dave Vellante, your next guest. I'm Del Fino, Senior Vice President, Sales and Systems Engineer and CUBE alum. Great to see you. Welcome back. Thanks guys. Good to be here again. We're here covering VMworld. The ecosystem has been a little tied, goes out, tied, comes in. Real clarity this year. Cloud, you want it on-prem, private cloud, your private cloud, or Amazon? Right. Any questions? Exactly. I mean, this is clear. This is the vision coming to fruition, right? This is what you're seeing this year at VMworld. I think, in particular, when you talk to the customers, they're now in a state of cloud reality, right? Or, you know, there was sort of this big rush, I'm going to try to move as much to the public cloud as possible. And then, you know, in terms of the scale they got there, they start to run the challenges on that side. And they realize, I need to have a dual strategy. I need to have a private, a public, a hybrid strategy. And I think you see all the announcements that we've made today with VMC on AWS, going live now, with more coming around the world in different zones as we progress throughout the rest of the year and into next year. And as well as, you know, all the service offerings we just announced, Wavefront as a service, VRNI as a service, NSX as a service, AppDefense, which is our latest security strategy as well. Customers really see how it comes together now, and they want to go down that journey with VMworld. It's important to clarify, I call the high-level messaging, so it's got clarity. But also, VMware and ecosystems has a lot under the hood, and it can get very technical. So you got to balance the speeds and feeds to feed the red meat to all the practitioners, and then the high-level. So I got to ask you the question. I mean, because the people that are sitting out there on this cloud reality that you mentioned, they don't have a lot of people sometimes. Someone's got to implement this stuff. Automation's coming. Okay, I get that, but getting to the cloud is not easy. I still got to run my shop. What is that operational reality right now? Because cloud reality, okay, I get it, but now I got to turn my on-premise into a true private cloud with a new operating model, new practices. How are the VMware customers dealing with that? You know, and I think that's part of moving away from the legacy as fast as you can, or at least where you have to keep it. You've got to sort of isolate it and put it in a corner, because it's the legacy that's holding most of us back, right, because I got to understand how to run the legacy, keep the lights on. That takes 20, 30, 40, 50% of my time, depending on the customer, depending on their infrastructure. At the same time, I've got to retool my skills. I've got to retool my toolkits that I use, my runbooks, my operational processes, but now they at least have a direction to build to. So all the customer meetings we're having here today, it's about software defined. How do I build this abstraction layer? Okay, we've been doing this with VMware for years on the compute side. Many of them have ventured down the journey with us on the NSX side. You see 10,000 customers roughly on the vSAN side as well. Now it's about putting it together, putting the automation tool set around it, and really building that same experience that they get in the public cloud, which it's fast and it's easy, on their on-prem data centers as well. And sometimes there's many reasons to retain things on the private side, data sovereignty, intellectual property, all of those things as well. So I think that's where the customers are on the journey right now, and it's now they feel comfortable with the direction and they're going to adopt quickly. I like this idea of cloud realities. We've been talking in theCUBE today about configuring the cloud to the realities of your organization's data. And you're talking about governance, security, data locality, et cetera. So it really comes back to a data challenge. You can't just take all your data and shove it up into the cloud. What are you seeing from customers in that regard? You know, and I think there's a regulatory component to that as well, particularly if you go overseas to Europe and Asia, there's a lot more challenges around that as well. I think what you're seeing is that customers recognize the fact that not everything is going to go into the public cloud at this point. So they're really prioritizing burstable workloads, temporary workloads, definitely a prime opportunity to put into public cloud. New application development, definitely a primary opportunity to put in the cloud, right? If I'm in the healthcare business and I have to retain healthcare records for X number of years, and I'm responsible for HIPAA compliance around them, maybe not something that I'm just going to shove up into the cloud today, right? So it's use case specific depending on, an application specific, depending on the vertical industry the customer resides in, and depending on where they are in their journey to the cloud as well. So the other, so you've got a lot of momentum in your business right now.  And so we talked about the cloud realities. That's part of it. The AWS announcement last year, even though it was a year ahead of time, gave a lot of clarity to people. How much of the momentum is due to those factors? Again, the cloud reality, the fact that people are now more comfortable with your cloud strategy and saying, okay, I'm willing to make maybe a multi-year commitment with VMware, is that a factor? It is a factor. It is a factor and I think the two remaining components accelerating and capturing momentum in the market of our SDDC strategy being VMware and being vSAN and NSX is also helped that reality come to fruition for customers as well, right? That this software-defined, we've been talking about software-defined data center for a long time, like everybody else in the industry talk about things sometimes a lot sooner than they come to fruition. But now that they put together vSphere with NSX with vSAN and they say, hey, I can actually build a private cloud that's fast and easy, which is the reason a lot of my IT people or my application developers were going around me because the public cloud was faster and easier. Wasn't necessarily cheaper, but it was definitely faster and easier. Now customers who've been on that journey with us for the last year, realize they can offer the same thing on-prem as well and take advantage of both. Does that make sense? Yeah, yeah. What's the biggest walk-away for you right now, looking at VMware, if you had to talk to customers that are not here and looking at the online coverage, certainly Twitter, you'll be able to do a lot of cube coverage and a lot of pictures, a lot of architectural slides. What's the big walk-away so far, day one?  In many different categories coming forward, you'll hear another big announcement tomorrow coming up in terms of what we'll be doing in conjunction with one of our sister companies in the application development world, but also about taking security to the next level with app defense, right? So microsegmentation has become fairly ubiquitously known within the industry now. How do I take that into the guest, into the operating system, into the application layer? How do I secure those things as well? You see a lot of customers getting hit ransomware attacks this year. Those are big reality checkers for you. If you're the one sitting behind the keyboard that's got to defend your environment against that and rebuild it, and I think they really see VMware continue to push the envelope to develop very innovative solutions to these approaches that are very cost-effective and that are also very high performance. Personal question, as you're out in the field talking to customers, you've been in the industry for a while, you've seen the waves. What's the biggest thing that you notice, observe out there right now? What's happening? Share some color with the landscape and the marketplace. I think there is some good recognition from customers around the type of operational transformation that they're going to have to go through in this journey. It's not about the network independent from storage, independent from security, independent from computer anymore. Infrastructure is one entity. That's the way the application owners and the application developers view it and want to consume it. That's the way that infrastructure teams are going to have to deliver it. So I think there's a lot of recognition of that. I think there's recognition that the security problem is bigger and better and worse than ever and it's not going away anytime soon and there's sort of no magic box, right? If there was, you'd pay a lot of money for it to make your problems go away, but it's really something that has to be ubiquitous. And infosec policy has to be aligned with infrastructure security implementation. I could have the greatest policy in the world if I can't actually implement it, I'm not going to get the benefits of that security there. So I think those are some of the things as well. I think sort of the container world is going through a little bit of the post-hype cycle, what's the reality check of that environment as well right now, you know? We saw this with OpenFlow and SDN five and six years ago. It's so expensive to run and why even do it, right? I mean, at some point it can be total cost ownership and ease of use in school topics. I think we're well into production ready phase of software defined networking. We're well into the production ready phase of software defined storage and hyper-converged infrastructure. You know, we need to take containers into that next phase as well. Bottom line, what does cloud ready mean to an enterprise these days? I mean, I think it means that cloud ready means that application, that workload is portable, right? And I can deliver the same level of availability, service and agility, whether it's in the public cloud or whether it's in my private data center, right? Or I move it back and forth between both. So, we're certainly excited about the momentum we see with our customers. I think you can see and hear the buzz around VMware going on this year. And I think it's the best it's been in a few years. You run the SE team as well. Yes, I do. How does that work? I mean, SEs are like the Navy SEALs, John, always talks about on the beach and they- I like to call them the conscience of the Salesforce there. Yeah, yeah, there you go. Yes.  Customer trusts them and, you know, and but the same time, they understand the customer requirements at a very, very deep level. How are they organized? How do they fit into the partner ecosystem? Maybe you could explain that a little bit. Yeah, so, you know, I think traditionally we've organized our SEs aligned them with product categories, right? So I've got networking and security SEs. I've got cloud management, automation, orchestration SEs and, you know, silver defined storage SEs. But I think, you know, that sort of is the baseline. And then you start to build their skill sets towards solution, right? Towards a solution. So what, you know, what types of solution is it? Containers on open stack. Is it VMware's SDDC stack? Is it around particular vertical solutions? So if you're an SE on my healthcare team, you're probably very focused on electronic health records and Epic and Metronica and, you know, different applications like that. And how do you solve those customers' problems at the higher level and be able to drill down at the same time with the domain experts from those customers when they want to understand how OSPF works in NSX, all right? Or how they want to understand how LUN creation works in VSAN. So it's sort of an evolution in terms of building skills. You got to start at the deepest levels and then you got to build to how those products and those technologies integrate together to provide the customer with a solution. So as you move toward this sort of multi-cloud word throwing another buzzword, but is this cloud architect like SE role emerging? Yeah, we'll call them a solution architect, right? And that solution may be a cloud solution. It may be a vertical solution targeted at a specific customer base. And make sure that we do what's appropriate to serve our customers in a generally good way. What's the coolest thing you worked on this year? I've got to think that app defense is the coolest thing that we've got out this year. I think that we've solved a lot of problems with micro segmentation from a network security perspective. I think now going up into the guest and into the application layer and providing an analogous functionality there is going to be really a very, very prevalent way of preventing breaches, malware, malware propagation, ransomware in the future as well. So I'm a little bit of a security geek. It's attractive to me. I really see that as just an ongoing, it's not even a battle anymore. It's a war now for our customers, right? And we want to help them win that way. Ransomware has been so brutal. Ransomware has been brutal.  Well, it's become a board level topic really overnight. Absolutely. It's a serious board level topic, not just lip service. I mean, you're seeing that, right? You will see in some circumstances, boards actually pulling the chief information security officer out of IT and having to report directly to the board. Well, I mean, it makes a lot of sense that, I mean... It's... The pressure is unbelievable. The pressure is unbelievable, right? In a lot of regards, you would think that the CISO should not, certainly should not report to the CIO. It's kind of like the Fox running the... Fox watching the hen house a little bit, you know? Dynamic, right? Maybe that's not the best analogy, but... Yeah. But there should be an inherent tension there. Number one, but number two is, what's the right regime? I mean, why is it IT's problem? It shouldn't be, right? Yeah, and I think it goes back to, you know, information security policy versus actual implementation. And the gap that's existed between those two for years for many reasons, networking being a flagrant issue in that context, right? Where I could say, oh, this application, this user needs to talk to this application, this application needs to talk to this set of data. How do I implement that? Right? That's not the easiest thing with the tool set that customers who run legacy networks have had historically. So I think now that we have some of those things, you know, you'll see the scenario I just described where some, a few organizations are pulling the CISO out of IT and reporting it to the board. Or some, you know, we've seen board-level mandates for segmentation initiatives within the technology area as well. So I think this is going to be an ongoing battle that we face moving forward. I mean, this is the biggest problem I would stand on the cue all day long because part of the value purposes of cloud and DevOps and apps is having data in real time. So to be liberal with the data, you run the risk of opening it up so you can't do it the old way. You know, part of the cloud adoption and the new way of applications about moving these businesses forward, right? The security is one of those things that will move you backwards from where you are today. Right? So I think that it's important that we be able to tackle all these battles on all different fronts at the same time. If I may, I know we've got to figure out but there's another dynamic as well which is the recognition that we are going to get penetrated. And I think it was the third leg of Pat's slide today was response. And so boards are saying it's not if it's when, how do we respond? That's a critical part of the implementation. And I think it's, you know, and it's particularly, we talk about IoT, right? Think about the number of new entry points you create into your infrastructure, right? Every device you connect to the network itself. So keeping them out is a huge challenge. The question is, what can you do as the owner or operator once they are inside, right? How do you limit, how do you restrict the level of risk that you have and exposure you have to your data, to your applications, to your customer information so on and so forth. And I think that's what we've brought the table in a substantial way with micro segmentation with NSX. And I think you'll see that continue to really, really raise the game when app defense as well. Don Delfino, great to have you, great color, great commentary. Man, you're like a pro. We need like a anchor with us. Sports center here. Can I have a job if Pat fires me, am I in? No, you're in. You're in. Pat fire him so we can hire him. Don't fire me, Pat. I like my job. Don, thanks so much. Great, great to be here. Thank you, Dave. Appreciate it. Appreciate it. Pleasure. Bringing a great attitude on the Cube, great energy. More come, day one as we continue down, wind down day one. Three days of wall-to-wall covers with the Cube, VMworld two sets, double barrel shotgun of content here in the Cube. We'll be back with more after this short break.


Hello, everyone, and welcome to Cloud Native Live, where we delve into the code behind Cloud Native. I'm Taylor Dullazal, head of Ecosystem here at the CNCF, where I work closely with teams as they navigate their Cloud Native journeys. Every week, we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things, they will break things, and they will answer your questions. In today's session, we have Andrew Reinhardt from Sedera Labs. And today, Andrew will be presenting Bringing the Edge Into Your Data Center. This is an official live stream of the CNCF, and as such, is subject to the CNCF code of conduct. Please don't add anything to the chat or questions that would be in violation of that code of conduct. Basically, please be respectful to all of your fellow participants and presenters. Be excellent to one another. With that, I would like to hand it over to Andrew to kick off today's presentation. With that, Andrew, please take it away. Cool. Thank you, Taylor. All right. So today, I kind of wanted to take us through a pattern that we're seeing emerge from our users, where, essentially, you have Edge Compute, but you don't have a whole lot of it. And when it comes to Kubernetes, you want to have HA and all these things. And it becomes much more expensive. And when things go wrong, things could go really wrong if your control plane is out there. And so what we're going to do today is we're going to deploy a control plane in Vulture. We're going to add some nodes to that cluster that are running in digital ocean. And then we're going to add a Raspberry Pi to the cluster that's running in my closet. And the goal is to run Redis on these worker nodes that are in digital ocean and prove that we can reach the service, the Kubernetes service, from the Raspberry Pi. So you can imagine that you have maybe a cluster that is hosting Argo CD or something or Flux that's responsible for deploying an application to your Edge machine, and then your Edge machine is reaching out to what people I'm finding out are calling cloud services. But they're not necessarily the AWS or GCP services. They are the cloud services that are running in Kubernetes, but the workloads are actually in the cloud and the Edge is reaching out to them. So let's see. Am I sharing my screen yet? I'm going to take a little bit of a shortcut today because we could have easily made this stream about how do you actually even go about connecting these remote machines? Because there's some networking implications here in this design that I'm talking about that make it very, very difficult. As I mentioned, we're going to have machines in Vulture. We're going to have machines in digital ocean. And we're going to have a completely private machine in my closet. So how does that work? WireGuard is a new technology that's built into the Linux kernel that we have support for in the Sedera Labs' product, Talos Linux. Talos Linux, it's just a simple Boolean value, you just say, enable kubespan. And all of this automated WireGuard networking gets set up for you. Didn't want to go through that manually. So I'm sort of cheating here a little bit. And I'm also using another product of ours called Omni, which is going to allow me to manage these machines regardless of where they're at. And what you're looking at here currently is about seven machines. We have these demo SFO three ones that are running in digital ocean. You can see these droplets here. We have these ones that have the generated hostname since Vulture doesn't do DHCP, it seems. But we do have three of these machines running in Los Angeles. I should point out that's in Los Angeles. This is in San Francisco and I'm in Santa Barbara. So those three machines will serve as the control plane. And then we have this demo, SBC01. Sounds fancy, but it's just a Raspberry Pi 4 running in my closet here. So let's start off by creating the cluster. I'm going to look for my, let's see, I want Talos, yes. So let's focus on the control plane nodes first. So I'm going to make these three in vulture.com, my control plane nodes. And I do have some patches here, which I'll explain as I'm doing it, but let me just look here, two seven, cool. So Talos Linux is completely driven by a configuration file that you supply to it via the API. For those of you that don't know what Talos Linux is, it's a Kubernetes or rather a Linux distribution that is built explicitly for running Kubernetes. You could only communicate with it via an API. We don't have Bash or SSH or anything like that. And so I'm going to use the API in this product to push this configuration to this machine when we're setting it up. And all I'm doing is setting up the networking according to how vulture.com tells me to. And you can see here, this coop span navel true that I was talking about earlier, this is all that is needed to do the automated wire guard setup. So let me just double check here. This is two seven one, I'm just matching them via UUID and I set up some patches here beforehand to save us some time. And really quick, Andrew, is there any way that I might be able to just bump up these sizes, just a hair or two? How's that? Good, that should be good for now. Cool, thank you. Yeah, and the last one is F6C. All right, cool. So we'll do tell us 120 and Kubernetes 124.4. We will call this live stream and I will create the cluster. What's really cool about this is that we are actually already leveraging wire guard. It isn't the coop span wire guard network, but it is what we call sedero link, which is basically Talos is streaming logs to this service for us. So I don't have Talos API access in theory at the moment. So if I wanted to see what was going on, I could get almost like a console access that you would from the cloud via this. And this is just Talos using the wire guard network. It's just a point to point connection so that we can stream events and logs and we can actually operate this machine remotely. You can see that there are some transient errors coming up, but the node is not ready, which means that things seem to be forming. We will wait for that to be formed. All right, cool. So the nodes are starting to pop up. These are the machines in vulture.com and this is going to serve as our control plane. And the way that we're doing load balancing here is that this product is actually serving up a load balancer so that we can hit all of these three machines in HA fashion. So as long as your edge machine is going to have egress capabilities and internet, you'll be able to reach the Kubernetes control plane and it'll all be secured over wire guard. Cool, so the next thing I wanna show is let's just add some machines here. And these machines are gonna serve as let's just imagine, we have a control plane currently running in Los Angeles and let's say that your customer has an edge site somewhere in San Francisco. So what we're seeing people do with this product and with Talus Linux is they're actually adding, I guess you can call them edge but they are still running in the cloud but they act as sort of a support set of nodes that supply different services that the actual really edge machine is going to need, the machine that's running behind the customer's firewall which in this case is gonna be my SBC-01 machine. But these SFO machines are just going to serve something. In our case today, we're gonna use Redis but just imagine that they supply some kind of support to the edge machine so that you don't need to run as much there. So we're gonna set up these three as workers and the only thing that we're gonna need to do on these machines is patch in the Coupespan capabilities. Cool. We'll add those nodes and all that we're left here now is with the edge machine. It is technically physically in Santa Barbara because I don't have an edge machine running in San Francisco somewhere but let's just imagine that this is the edge machine close to these support worker nodes. Before we do that, let's just make sure that these other machines join in. I always wish that Projects supported the Jeopardy theme song music while they write for things to play around join but no one's answered that feature request yet, sadly. All right, cool. Node is not ready. All right, it looks like things are going to chug along just fine. So one of the things that you need to be concerned about that we're seeing our users at least be concerned about at the edge is how do you encrypt the disk and thankfully this is simple enough of Talos. So we're going to add this machine as well now but we're going to do something slightly different with it this time. We're going to patch in, we're going to patch in some encryption. So we're going to use Lux to encrypt the what we call the system disk within Talos which is the disk where more or less ephemeral state lives. We could do a whole live stream on Talos but I won't get into that but just suffice to say we're going to be encrypting all of the important partitions that we need to be worried about. And what I'll do here is just generate a couple passwords. So this is something that you probably want to be thinking about if you're doing an edge machine because people can walk away with these machines and that's a common thing that does happen. So you can see we have kubespan enabled and we're going to do system disk encryption and I'm going to add that node. So this machine running in my closet should be joining this cluster here very, very shortly and it looks like the ones in San Francisco are up. Let's just double check in San Francisco if we have some extra disks because if time permits I want to maybe push things a little bit and try to get storage going because that is a common thing that does come up is how do you do storage? To be honest, I think that that is a weakness in this space and I think these hybrid sort of approaches are rare because it's been hard. But I think if you look at WireGuard and what it has enabled just take the folks at tail scale what they're doing. The new types of things that we're going to see I think are going to push the boundaries of the current implementations of things like storage. And so my hope is in the future we can have storage that is able to be sort of geographically aware. But yeah, it's not really the case today. So the recommendation I would say is sort of the pattern that we're doing here today is where we're going to have Redis let that be backed by a local sort of storage some CSI that you decide to supply to the cluster and then have your edge machine sort of store its data there if you will. Okay, cool. So actually let me just double check on this one machine real quick because it is important that it works. No, it isn't ready. All right, cool. It looks like it's going to come up though. All right, let's download our Kube config and let's download our Talos config. These are going to allow us to actually interact with these machines and I will move these where I need to go. And I'm happy to pause for any questions at this point because we're going to sort of shift. We've bootstrapped this cluster and I've already explained sort of the architecture that we're seeing people use the pattern at least and now we're going to actually prove that this can actually be functional. Cool, if you do have any questions please feel free to throw those into the chats and we'll get those questions raised up. And then we had another request just to make the textures a little bit larger in the terminal. That better? Yeah, just a little bit more would be fantastic. Perfect. All right, awesome.  All right, so let's just do get nodes and show that things are working. Awesome. We did just get one question asking, are there any plans to release this for air-gapped environments? Yes, absolutely. It is a single go binary. It's going to have SED baked in so you'll be able to run this in HA fashion. So yeah, it will be coming shortly but at the moment it is currently just a SAS. Cool. Cool, so we have all of our machines and at this point I'm going to do some labeling on these ones that are running in digital ocean so that we can do some node affinity. So this is something that you're going to have to be really aware of when you're talking about the edge in this pattern because imagine that this edge worker, this SBC-01 here, let's imagine we have five of them and they are different customers potentially or different locations. And so you're going to have to really utilize labels and node affinity to make sure that you have the workloads running where you need them to run. I'm going to use the node affinity today to make sure Redis is running in my digital ocean machines but you could see how it applies to these edge machines as well. So if we get our nodes now, we can see that these ones now have a role of Redis arbitrary name but something that I want to also point out which is interesting here. Oh, the internal IP is that which is interesting. You can see another thing that you're going to have to probably be very aware of is sort of networking collision. In Vulture, I have my subnet is 10-5-9-6-0 slash, it's a slash 20 and then in digital ocean, it's 10-1-2-4-0-0 and it's a slash 20 and at home it's a 192-168-1-1 slash 24. So you got to be aware that you're not going to get networking collisions here because what's going to happen under the hood with at least Talos and I think anything that isn't necessarily Talos but it's going to implement something equivalent, you're going to have some potential for IP collisions here. So that is something you're going to have to be aware of in this model as well. Let's just double check. Actually, I need to get an IP address. So let's just ask this machine what version it's running just to make sure that we can have Talos access. Ah, I know what I need to do. I need to set the password on the Talos config and I will do that offline. So give me a second and I'm happy. If there's any questions as well, this is probably a good time to take it as I do some stuff off screen here. Awesome. If anyone has any questions, please feel free to get those to us and can definitely ask. I think when it comes to, I really liked what you said about kind of envisioning what the future is going to look like in terms of where we actually go about placing storage. Are there any other projects or contexts or really just any places to be aware of where some of those conversations are happening that you could refer folks to, keep an eye on if they're interested in that space. Yeah, they're not really happening yet. I think what we're doing here at Sedora Labs is sort of pushing the boundaries. I think with something like Talos Linux, we can really start to imagine new types of architectures and new ways of doing things that just aren't there yet. So we're a little bit ahead of the curve here, I think, and the conversations aren't happening, but we plan on spearheading those conversations. But they need to happen in multiple places. Storage is one. If you even just look at the ecosystem within Kubernetes, they don't really support this notion of hybrid infrastructure, if you will. This idea, like what we're doing today, I have machines that are running in Vulture, I have machines that are running in Digital Ocean, I have a machine that's running on-premise. If you look at tooling, they don't really, you know, cloud controller managers, for example, they don't have the notion of this. How do you actually do that? Look at cluster API. It's not a native idea within the cluster API world. So we're still very early on here, to be honest, and the conversations aren't necessarily happening at the level that I think we should, but we're proving it out here, that it can be done, and we plan on driving those conversations. Cool, cool. It's, yeah, it's an exciting space. I'm really curious to see as we move to the edge, you know, what is unlocked? I know there's still many other problems to solve too, like multi-cluster federation, like all of those kinds of concerns, as well as we start to go multi-cloud and multi-context on that front. Exactly. I did get one more question in, and that was does latency between workers and the control plane matter? Not too much. What does matter is the latency between at CD. That is the most important thing. And so in this model, you could see that I'm deploying the control plane nodes right next to each other. That's important. The amount of traffic that's going back and forth between the Kubelet and the API server is pretty minimal. And it really is just enough to really check the state of the world and make sure that the state of the world, the configuration is pushed to the machine. And so it's not necessarily in the hot path as far as latency goes. So that is actually a very, very common question. I'm glad someone asked that. Let's figure out why this isn't working, the joys of live demos. Okay, looks like I'm gonna get another IO time out. That's fine. We can live without it. I would have loved to have shown off Talos CTL a little bit, but we can figure that out later. In the meantime, let's just actually start crafting some of the things that we're going to need for deploying Redis. So I will open up my editor here and let's just call this redis.yaml. So my idea here is that we're going to run Redis as a deployment. We're going to target the nodes that have the role Redis. Those are running in digital ocean. We're gonna deploy Redis there. And then we're going to expose an internal service which is going to be routable from the edge machine thanks to WireGuard or Kubespan. So that we can reach this service from, even though this machine is completely private, we'll be able to reach this and prove that this is a model that will work for us. So I think what we'll do is, how are we doing on time? We have about 40 minutes. Okay, cool. So I know that Kubernetes has, the reason I chose this is because I know Kubernetes has a sort of a how to run Redis. And so we're just gonna work from there to show how we can build this up from scratch. So let's just start with the pod. I know that what I wanna do is actually do a deployment. So we'll just change this. Vaps v1. And then I believe what we need is spec. I'm going to cheat just a little bit. So we'll just do one replica for now because you could figure out how to make this HA later if we have time, maybe that's what we'll do. But I'm just gonna use this label here to set up my template. And then let's get these things going. I think that's what we need. And deployment, cool. So I mentioned node affinity. So node affinity, again, it's going to be an important idea here and you're gonna make sure that you have these on all your workloads when it comes to this model. What we're going to do is we're going to set up some node affinity here. And I'm keying in on this node role label that we did earlier and the values are just that. That's not really the node role. This is sort of a special one, by the way. You cannot, the Kubla does have the ability to set labels on nodes when they come up, but this is actually not allowed because you can actually escalate a machine to become a control plane node using just this label. So they don't allow this to be set. So this labeling will have to be done sort of after the fact. But anyways, here's the affinity. We're going to target those digital ocean machines and we're going to run Redis. You can see that we're referencing a config map here. I believe they have that somewhere around here. So we'll just use this one as an example. Cool, so that will be mounted up. We have port six, three, seven, nine. And the last thing that we're going to need is a service. So I want to be able to reach this service from my edge machine. So let me just copy this one in. Save us a little bit of time because no one wants to watch me be a YAML engineer. Cool, so let's try to deploy this. What is the name of it, Redis, YAML? We might get yelled at about something. Yeah, okay. These are just warnings at the moment. Let's see where they are running. Cool, so it has reached one of my digital ocean nodes in San Francisco. So we have that running, great. Now, what I want to do is I actually want to hop in and I want to create a workload that's going to simulate an application that's running at the edge on this SBC01 machine. So let's just create a pod here. We'll call it Redis Client. You can imagine this is your application or whatever might need to reach this service that we just deployed in the cloud but that we needed at the edge. So I'm using a different one here. I'm explicitly setting here. So you can see here, I just did node affinity. This is a good way of just targeting node types, if you will. In this case, we're just using the role and you could even have it be such that if I scale this up, I only want them to be ran one on each machine. There's ways to spread this out. In this case, it's very, very simple. I'm just gonna say, okay, I know that this is my application, my workload that I want to run on this specific box. So we're going to target that specific box by hostname here.  It's going to apply Stig Hardening guidelines, CIS benchmarking guidelines, our own sort of personal opinions and the on things. It's going to be a hardened version of Kubernetes, which I highly recommend for the edge and this pattern as well because there's just so many moving parts in this. And so that's where all of this extra security context stuff comes from. It's basically the most least privileged pod you can possibly run. And the only thing that we want to do is exec into it. What is the name of it? Let's just show that it's running on. Cool. So my client and my server and let's reach it. Everything is ready. What could I be missing? Interesting. CoupsPen should be enabled. Okay, well, let's dig into it. Let's use Talos CTO. I believe we could use it here. Let's get the members here. So we'll use Talos CTO to sort of debug what's going on here. So what I'm doing here is I'm just going to get a list of the resource definitions. Talos, if you haven't noticed already is very, very similar to Kubernetes. It operates on this controller pattern. We have resource definitions that you can supply and that you can retrieve. I'm going to look at my CoupsPen peer statuses. What I'm doing is I'm targeting one of the machines that I can reach. CoupsPen is not enabled here. What is it going on? We'll look at the configuration. Oh, interesting. Did I not enable it on these ones? Let's double check. That's interesting. Oh, it looks like my patches didn't apply for some reason. Did I not save them? Well, let's just go through and let's do it. The other things don't necessarily matter. What I will do, let's get these nodes. So what I'm doing here is these machines for one reason or another, I'm not entirely sure about right now. They didn't get the patches that I supplied. So the most important one is that we enable CoupsPen. So I'll just start with, well, five eight since I already have it here and we'll go down to network. This is the Talos's configuration. We'll do CoupsPen. That should get us what we need. Give that a second. I think it needs 30 seconds or so. We'll edit the other machines in the meantime. So I have 180 and 94 to do. Feel free to interrupt me if there's any questions at this point. Yeah, but not a question, but a comment. The excitement of a live demo. So... But that's what makes it fun and that's why we do it. I can't wait to endorse you for debugging live on LinkedIn. I feel like that's one of the most undervalued skills you could possibly have. Operating under pressure is one of the most important skills in that case. Helpful learning, every adage. Yeah. All right, cool. So we're getting some peers at this point. I wonder why I had some patches that didn't go in here. Let's do a little bit of debugging real quick. Okay, so this one has peers now. The problem is going to be that I can't reach what we can do. Let's get the public IPs of our droplets and double check. Interesting, okay. It looks like none of the patches worked. Well, they didn't apply for some reason or other. I believe we just rolled out an update to this alpha software just before I did this. So we'll have to check on that. But we'll copy this, maybe some hidden character or something. Okay, I guess so. All right, we applied that configuration, which means that should start setting up Coupes Band in the background. We'll do this one as well in digital ocean and then we'll just edit the local one here in Santa Barbara and then we should be able to continue. But hopefully you can see a little bit about Telos and how it works here. Cool. Yes, helpful to get a sense. I did have one question and someone that just really wanted a recap on some of the edge devices that you had just on clouds and edges. Oh, sure. So what we're running here is a seven node cluster. It is split across vulture.com where I have these machines running as my control plane. And then we have three of them. These ones are in Los Angeles, by the way. We also have three droplets running in digital ocean. These are running in San Francisco. These will serve as sort of the support nodes, if you will, for our edge machine. And our edge machine is this SBC-01 machine, which is running in my closet on a private network here at home that is just a Raspberry Pi 4. So let's enable CubeSpan on the last machine here. This one. Let's try one thing. Let's actually... Well, we may need to just improvise here and spin up something else that would represent our edge machine here. One question that came in was, what are the advantages of CubeSpan over, for example, Cilium cluster mesh? Yeah, that's a good question. I think that they both are after the same end goal, but the advantages within Talos is that it starts very early. It's not dependent on Kubernetes status. It's not dependent on the KubeLit being up. This is just baked in and you get this as early and as possible. So we also have access to host-level networking routing. So we can set up routes and everything that we need so that even host-level traffic can be over this WireGuard network. So that is some of the advantages and it's being leveraged here already, but in this Omni product that I'm using, which would not be possible because if you think about it, these edge machines and the fact that I can remotely manage them using this product is entirely dependent on the fact that Kubernetes is not in the hot path because if I had to bootstrap Kubernetes on the machine and do all these things, it implies that I need to be able to talk to that machine and we need to get Cilium installed and do all of that. In this case, it's just baked into Talos and so it can start as soon as the machine turns on, essentially. Okay, so I think what we're going to run into here well, an issue. I need to, oh, I can schedule it, but I can't reach it. Well, let's just do something. Let's just create a droplet in somewhere else. I think I only have this in a few regions, so let's do this. We will do basic and we will make this simple machine and why not? Let's do New York. So what I'm going to do here is I'm going to spin up a machine in New York, in digital ocean. It's now going to represent my Raspberry Pi. I confirmed with my team in the background that patches were broken between my testing and now, but that's why we do this and that's why it's still alpha. So we'll still be able to prove this out. What I'm doing is I'm booting a custom image that you can download from here. So if I just go to here and I'd say digital ocean, I could download this image and it's preconfigured to set up WireGuard so that we can reach this machine and start managing it from here. In fact, it's already joined. This is the one in New York right now. And what we'll do is we will join this to the cluster. I can fake patching, that's fine. I'll have a public network and I can patch it in manually. So let's add that node and then what we'll do in the background here while we wait is I now need to, what's the host name of this machine? I believe it was demo NYC 3, 0, 0 1. Let's delete that pod. We'll get it off of that machine. Cool, so we still have the one in the service Cisco. Let's just check in on demo NYC 0, 1. No, it isn't ready, but it should be there. Okay, all right, there's NYC. We have an internal, we have that. So we're gonna patch in a coop span here so that we can have this mesh network. And if the node is ready, we can continue. So all I've done is I've just spun up a new machine. This might actually be good because you could see how things could go wrong and maybe you could just spin up another machine and replace it really quickly. This is why we love Kubernetes. So we have that pod, let's exec into that. We'll use the service to see if we have it. Cool, so this is a node running in New York, just a recap. We have our control playing running in Los Angeles. And we do that because we want XEDs, the latency between XED to be as minimal as possible. Thanks to WireGuard and coop span, we're able to add some machines that are running in San Francisco, which would act as these support machines offering storage or something to that effect, some service that the edge machine might need. We planned on using the one in a Raspberry Pi in my closet but we ran into some trouble. So to replace sort of what is representing that, I just went ahead and spun up a node that is running in New York and using WireGuard we're able to have a completely mesh network between all of these. So you can imagine maybe that this is one customer and this is another customer and you have workloads dedicated to those machines. Maybe it's audio visual and you need to run some software in a conference room that supplies that service. And it's a room that is completely represented by a single machine. You can target that and run specifically what the customer might need. So that's where I wanted to get to with sort of cheating but we didn't necessarily cheat. We had to do some workarounds and patch things in manually. And then if we had enough time, which we might, I was going to maybe look at how we can maybe use Argo CD or Flex to deploy this or maybe even challenge this thing of storage running at the edge. Let's see, we have about 10 minutes. Are there any questions? Anything that I can answer? I don't see any questions now but folks if you do have any questions or would like to get things answered, please feel free to throw this into chat and we can get those surface. Cool. So I want to just maybe expand on this a little bit. There is another pattern to this that you could tell I was doing everything sort of by hand with Coop CTL and applying these. But in a fully automated, the goal of being fully automated, we're starting to also see that things like Argo CD and Flex become really, really powerful here. And one of the ways that we can deploy that stack automatically is with Talos. We support the ability to supply manifests at the bootstrapping time of the cluster. So that means that deploying your clusters could be as simple as what we did today. We just supplied patches that might do specific networking that I might need, enabling Coopspan if we need to. And then having your workloads and using these node affinities and all these things be managed by Flux. I've heard really good things about Flux and I did save Flux to be completely new to me so that you guys could really watch me stumble with this because I was told that's good for the live stream. So I kind of want to take a shot at seeing how far we can get in the next 10 minutes or so in automating this Redis client and this redis.yaml with Flux. I've used Argo CD before, I am a fan of it. I've never had a chance to run Flux so I'm gonna use this to do that today. All right, let's see how easy these docs are, get started. While that's going, let's see. Okay, so we're just gonna do some prerequisite checks. Bootstrap, I'm gonna need a repository. So let's create one. We did have one more question about Silium and Cubespan. This question was, any benefits of putting Silium on top of the provided Cubespan or dangerous disadvantages? We did, so we did have a bug recently with Silium and Cubespan being enabled but I believe we fixed that. I probably wouldn't run Silium with the WireGuard network, with this WireGuard network as well because we're setting up some routing rules and that will too, it's just untested. I'm not necessarily opposed to the idea and theory but I just think it's untested. Silium and Talos is a very common combination, maybe partly due to the fact that both are sort of more advanced technologically and are more cutting edge, if you will. And those types of folks tend to want to run those things together. But it is a very, very powerful combination. I'm excited about what Silium is doing for sure. And then the other question was, is it your own Raspberry Pi? It is. I managed to get one before the big shortage in them. So I'm gonna take this off of the screen real quick so I can set up my GitHub token. You can see my terminal still. Here you go. Cool. Let's do flux check pre. Pre-wreckers, it's past, let's bootstrap. I have no idea what any of these mean but let's see, I like to see stuff fail. Flux demo. That's actually a really nice experience. Good job, Flux. So what I will do is I will add things to this now and see if we could push it up and roll it out. I'll delete them first actually too. Okay, so Flux is gonna create this. I'm assuming this is a namespace, but let's see. That make sense to you? I mean, looking at Flux demo, clusters, my cluster and then these look like namespaces, I guess. Let's see. I never read all the way through. I just want to run commands and see stuff, work or break. How much time? We have five minutes, let's see. Okay, cool. Is this done bootstrapping? What is going on? Waiting for customization to be reconciled. Okay. I really wish all showed everything. I think it doesn't. I think I've got a couple GitHub gists of how to clear out like errors and things like that across namespaces and whatnot but I feel like, yeah, that's definitely, definitely could benefit from having something like that. Yeah, I'm wondering, when it says all, I want everything but I think they actually don't list everything. Cool, a couple more minutes and yeah, we won't be able to get that far I guess, but let's see. There was a question, once deployed, is there a UI for performance in management? Not sure if that's in reference to Flux but Steve Francis, but yeah, let us know on that front but I know with, I think with Flux, it's just that using the CLI and then I'm not sure with the other stuff, I'd assume that that is the case. Right, Andrew, as far as the SaaS and everything goes with Cidero Labs. I'm sorry, the question was again, sorry. Once deployed, is there a UI for performance in management? A UI for performance in management. I mean, it depends on what you're talking about. I don't know if Flux does but I know that we have some stuff here. So if we go into this cluster, we click on a node for example, or this is just the cluster view. I would recommend deploying something like Prometheus and Grafana to do full on metrics that you can alert off of and whatnot but this can be useful just to see what's going on in bootstrapping. If we wanted to look at a node in particular, this is using all of the Telus Linux APIs at this point. So we're able to get a list of processes, a list of services. If I wanted to look at NCD logs, I could do that here. If I wanted to get some lightweight metrics, this is all just GRPC streams. So looking at a list of processes and how busy they are and how they are over time console. So hopefully that answers the question. And we can dig into pods here as well. We do intend on expanding the functionality here but it is focused on cluster management at the moment. Well, yeah, I don't think I'll be able to get too far on flux. It looks like it failed anyways and I'll have to dig into that but I was prepared for that. This was something I wanted to test with just a few minutes left. But yeah, I think that that's it. So just to summarize, we're using Telus Linux and CoupSpan, which leverages wire guard under the hood to connect machines that are on disparate networks so that we can have a fully meshed Kubernetes cluster where the control plane is running in one region of the world. I can have sort of a support workload running close to my edge machines and my edge machines then can be smaller, I need less of them and less things can go wrong. So these edge machines will be able to leverage the wire guard network so that they can reach internal Kubernetes services regardless of where they are. Awesome. Cool. Should I stop sharing? Yeah, let me do that so we don't have infinite mirror. Infinity is a fun number though. Awesome, awesome. Well, thank you so much for stepping by Cloud Native Live today, Andrew. It was really great to kind of get a sense for how we can expand the edge, every cloud is a silver lining and it seems like that's the edge, the more that we get to see it. So thank you so much for joining today. Thank you for having me, that was fun. Thank you to all of you viewing today or watching this a little bit later for joining the latest episode of Cloud Native Live. We really enjoyed the interaction and all the questions that y'all had to ask. Thanks for joining us today and we hope to see you again soon with that. Wish you well and hope you have a wonderful week. Thank y'all so much, have a good one.