Hi everybody, this is what we learned from the Gateway API, designing Lincredi's new policy CRD. This room is so big, I don't know where to look. So this is a talk by my colleague, Matei David. He was not able to make it, unfortunately, because of visa issues, so I'm giving this talk instead of him, so hopefully I do a good job with it. So my name is Alex, I'm a software engineer at Boyant. We're the creators of Lincredi. I've been a Lincredi maintainer since the beginning of the project, and it's something I'm really passionate about. I think service meshes are very, very cool, and we have recently done a lot of really interesting stuff with the Gateway API that I want to talk about. Okay, so show of hands, who is familiar with Lincredi here? Okay, a lot of people, great. What about who's familiar with the Gateway API? Still a good number, but fewer, okay, cool. So I guess I don't have to spend too much time talking about Lincredi itself. It's a service mesh, it's a graduated project in the CNCF, it's used in production in many different places by many different people, and it's a very collaborative and active open source community. So as a service mesh, that means that there is a sidecar proxy in every pod that's part of the mesh that adds a lot of really interesting functionality like MTLS between all your services, reliability features like retries and timeouts, and a lot of observability features like layer seven metrics on success rate, request rate, and latency. And with a major focus on operational simplicity. So the whole thing kind of works out of the box and doesn't require you dedicating a lot of brain power to making it work. So the main focuses behind Lincredi are just that it's supposed to be very, very lightweight both in terms of resource usage, which means that it doesn't take up a lot of memory and it doesn't add a lot of latency, but also lightweight conceptually in that it's a very simple model that you don't have to think too much about and you can operate it without getting bogged down in the details. So being simple and secure right out of the box has been kind of a guiding principle for the project. The control plane is written in, this says go, but it's actually now a combination of go and rust, and the data plane is a custom built proxy written in rust built to be ultralight.  So this is a fairly new feature that was added in Lincredi 2.11, I think sometime last, earlier this year or late last year maybe. And that's kind of set the stage for why we care, but the gateway API and how it's gonna help us. So the idea is that we wanted to add a way to do authorization in Lincredi. So to take a step back, Lincredi has MTLS and it's had MTLS for a very long time. And what that means is that for every pod in your service mesh, it has an identity and it has this workload identity that has been built up from its service account token and that gets transformed into a certificate which it uses for all of its communication to other pods. And so that means that when two pods talk to each other, the M and MTLS means that they can mutually authenticate their identity. So if I'm talking, if I'm the foo pod and I'm talking to the bar pod, I can cryptographically know that the person on the other side of this connection is someone with the bar identity and likewise they know that what's on my side of the connection is someone with the foo identity. And this is really awesome because it means not only is that connection encrypted, but it's also we know who the other person is and we know that securely. And so we've had that in Lincredi for a really long time but what we haven't had is really a way to act on that information, a way to do authorization which is to say, I'm only going to allow certain identities to talk to me or I'm only gonna authorize certain parties from establishing connections. And so we wanted to add that and that was a major focus of Lincredi 211. And so as we were kind of designing that feature, one of the things we really had to think about is well, where does that configuration live? And as I mentioned on one of the earlier slides, one of the guiding principles of Lincredi is to kind of minimize configuration, make it easy to use, make it just work out of the box. But there's a few places where you really do need user input in order to decide what the behavior should be. We can't automatically determine who should be allowed to talk to whom. That's really something that has to be configured because only the people operating the cluster know what that behavior is supposed to be. And so this began the thought process of where should we put this configuration? And a natural place to put that configuration would be on the service, right? When we think about services in Kubernetes, we think, hey, I want to restrict access to this service to only certain identities. So that seems like a natural place to put that configuration to somehow attach that authorization policy onto a service. But as we kind of got into this, what we realized is that service might not actually be the best place for this. And the reason that is, is because when you think of service, there's actually kind of two concepts that are sometimes co-mingled there. I think the Gateway API kind of refers to them as service front ends and service back ends. I like to think of them as service targets and service receivers. And so what I mean by this is that the service kind of has two parts. So there's the service front end or the service target, which is usually a cluster IP or a DNS record. And that is something that a client sends traffic to, so it's a target. On the other hand, there's the place where the traffic actually goes after it is sent to that service. So this is usually a list of end points or a list of pods or a list of back ends. These are sort of where the service goes or this is where the traffic goes when you send it to this service. And a service object kind of co-mingles those two concepts. And if you wanted to attach policy onto that, authorization policy, you run into this problem where you can have pods, back end pods, which are in multiple services. You can have back end pods, which are not in any service. And so when a back end pod receives traffic, it really doesn't know which service was used, which service was targeted to send traffic to me. So in other words, if we had authorization policy attached to these services, we wouldn't know which policy to use. And furthermore, if a client connected directly to that pod without using a service, does that mean that it should bypass the authorization policy? So of course, this doesn't make sense. So we had to come up with a different resource to kind of encapsulate this idea of a traffic receiver rather than traffic target. In other words, put this authorization policy directly on the service back end, not on the service front end. So we came up with these new resources, server and server authorization. And if we take a look at what's here in the server resource, it's very, very simple. So there's just a pod selector, which selects which pods this service refers to, or this server, I'm sorry, this server refers to, and a port. And so this is another difference from services where a service can have many ports defined and you can actually use a service and target a port that's not defined on that service. With a server, it's specifically talking about one single port. And so this defines very specifically a traffic receiver. And so therefore we can set authorization policy on that server. And we can say, well, for this server, here are the clients who I want to authorize, who I think should be allowed to talk to the server. And so that's what the server authorization resource is. It's a resource which selects which server it's gonna apply to, and then it's gonna give a list of clients that are allowed to connect to it. And that list of clients can be defined in a number of ways. You can say I'm gonna allow anyone, just unauthenticated and anyone's allowed to connect to it. Or you can say anyone, as long as they're in the mesh and they have a valid identity they're allowed, or you can restrict it down and say only these certain identities are allowed. And so those two together make up the linker de authorization primitives. So this is kind of taking a step back what that kind of looks like. So you see on the right there, there's the bar pod and that's the one we're trying to control access to. Above it you can see that there's a server that's defined for it, which selects a certain port. So in this case, we're just talking about port 80. And then above that, there's the server authorization which says here are the clients which are allowed to connect to the server. And in this case, we're only gonna allow MTLS connections from pods that have the foo service account identity. And then so on the left hand of the slide there you can see that on the top there's the baz pod which does not have the right identity so it's not allowed to connect and that connection will be rejected by linker de. And then at the bottom the foo pod which is allowed because it does have the right identity. So it's allowed to connect. So what does this have to do with the gateway API?  So the gateway API is the set of Kubernetes resources that are very useful for defining the behavior of gateways. And one of the kind of key ideas here is that there are a bunch of different resources which exist at different layers and are owned by different personas. So at the top there you see there's the gateway class which is a resource representing a type of Ingress or a type of gateway rather. And then below that you have the gateway resource itself and so these might be owned by cluster operators. And then below that you've got HTTP routes which can be owned by individual application developers and those attach up to the gateways and say here for this route, I'd like to attach this to the gateway and if you get any traffic for this route please send it to the service. Here's another route. If you get any traffic for this route please send it to this other service. And so there's kind of some cool ideas in here. This idea that you can have these resources at different layers which kind of attach up to each other and that they can be owned by different personas. So this is kind of how that looks as an Ingress is as Ingress traffic is coming in. That traffic first goes to the gateway and then it's going to look and see which HTTP routes it has attached to it and whichever route matches that traffic is going to determine where that request is gonna go to. So what does this have to do with service measures and how is this useful? So this is all kind of to do with Ingress which is not something that LinkerD really does right now. LinkerD is more interested in managing east-west service to service traffic within the cluster. So how does this apply? And if you take a look at the structure of the HTTP route resource there's a few kind of interesting parts to it. So on the top right you'll see there's a parent ref on an HTTP route and this is kind of describes what that route attaches to. So in all of the Ingress cases we had that attaching up to a gateway but you could imagine that attaching to someone else something else. You could imagine having an HTTP route which is attached to a server or a service in your cluster and defining policy for service mesh traffic rather than for Ingress traffic. And then down at the bottom you see that it's kind of made up of two parts down there. There's the match which defines what kind of traffic is gonna match that rule. So that's usually path, either path based matching or header based matching or a combination. And then on the right you have the behaviors that should be taken for traffic which matches that route. So either a set of filters that apply some logic or some back end refs that indicate where that traffic should go. And so this is kind of an example of what that spec might look like in YAML. And so the kind of idea that we really latched onto here which we thought was really interesting is well one this gives us a way to talk about routes which allows us to specify policy in a more fine grained way. It means that when we're talking about authorization policy we don't have to authorize an entire server and say for this server here are the clients that are allowed to talk to it. We can be a little bit more specific and say well we actually only wanna authorize these clients for this route on this server. We wanna say that if for example you have a admin server you might want unauthenticated access to the liveness and readiness probe endpoints because that's gonna be hit by the cooblet. You might wanna make sure that Prometheus has access to scrape the metrics endpoint. You might want to further lock down other endpoints that are let people do administrative tasks and make sure that those are only accessible to people who really should have access to them or service accounts that should really have access to them. And the other interesting thing here is that this gives us a way to kind of attach not just authorization policies but potentially other kinds of policies as well onto these resources. So this is kind of the structure that was informed by that in Lincardy. We adopted the HTTP route type from the Gateway API. We had to modify it a little bit in order to fit our purposes but we now have support for HTTP routes and those can attach onto servers so that you can authorize routes rather than authorizing an entire service. So it gives you that more fine-grained control. And we also made this a little bit more generic so that when you have an authorization policy that can kind of target a wide range of different resources depending on how granular you want that authorization to be. So you can authorize access to an entire namespace just to a specific resource or even more specifically to a specific route. And so this was added in Lincardy 2.12 which was released fairly recently so we now have much more granular support for the way you can do server authorization. Okay, so looking to the future, what's coming next? So if anyone is kind of familiar with some of the more advanced features in Lincardy, one of them is called service profiles. And so service profiles are a feature we have in Lincardy that allow you to configure things like retries, retry budgets, timeouts, and do that kind of on a per route in a per route way. So you can say these routes are retryable, these routes have timeouts. And that's gonna sound pretty similar to the stuff that we've been talking about earlier which is attaching policy onto HTTP routes defined in the Gateway API. So these are kind of two parallel ways of doing the same thing and we want to kind of slowly unify that and move away from service profiles and move more towards this Gateway API style approach of policy attachment where you can have resources which represent policies like retryability or timeouts or authorization and you can attach those policies onto HTTP routes. And this gives us a much more, an approach that's a lot more consistent with the way things are done in the Gateway API and feels more Kubernetes native than what we had before. So why are we doing this? What is the purpose of adopting the Gateway API? I think there's a few reasons of why the Gateway API is really interesting for defining these types of things. One of the primary ones is that these Gateway API types give us a standard way to define this rather than having to come up with these new structures or new types ourselves. So rather than needing to maintain our own version of HTTP route or route, we can rely on what already exists in the Gateway API and that can kind of be maintained independently of our, of, of Linkerty and we know that it's, you know, it's gonna be kept up to date. So that's a lot easier for us as maintainers. It's a lot, also a lot easier for adopters who are looking to use these technologies. It means that if they are familiar with the Gateway API, they've learned Kubernetes and they're trying to adopt Linkerty, they don't have to go and learn an entirely new set of APIs and an entirely new set of resources and an entirely new set of concepts. They can just use what they already know and it should feel intuitive and natural. And the Gateway API is just a well-designed API. It's very well thought out. It's got extensibility points built into it in places that make sense. And it's, it's really nice to work with. It's very intuitive. The cons of adopting the Gateway API are that it's still fairly new and it's still kind of evolving a little bit. So there may be still things that we don't know or that we discover over time and there's always a possibility for API churn as it continues to mature. But I think being part of that conversation and adopting it now, I think makes a lot of sense for the project. So I'm very excited to see where it goes. And in fact, you know, the Gateway API was originally intended to be used for gateways. But there's an initiative called GAMA, which stands for the Gateway API for Mesh Management and Administration, which is working through how the Gateway API should be evolved to accommodate service mesh use cases. So we're very involved in these discussions. If this is something that's of interest to you, I highly recommend that you get involved as well. And kind of as these discussions continue to happen and we continue to work on this, we're going to be at the forefront of supporting whatever GAMA recommends for using the Gateway API in a service mesh context. Okay, so what are the main ideas here? I think the main things we learned from kind of working with the Gateway API and trying to adopt it and use it for linkerty is that this difference between traffic front ends and service front ends and service back ends was really important. Trying to reason about the difference between a traffic target, in the case of a service, this is something like a cluster IP or a DNS record and a traffic backend or receiver, which is the actual endpoint or pod that's receiving that traffic. It's really important to keep that difference distinct in your head, otherwise things can get very, very muddy. And I think the Gateway API does a really good job of calling out that difference explicitly. The other idea that we really wanted to lean into is to not reinvent the wheel. We wanted to use the Gateway APIs because they made sense for us and it didn't make sense for us to just kind of come up with those APIs on our own when we could be engaging in the community and kind of playing nice in the ecosystem. And this policy attachment idea from the Gateway API where you can have these policy resources that represent things like client side policy, like routing or circuit breaking or retries or timeouts or server authorization and having those attached onto HTTP routes or onto higher level resources is a really nice framework and it makes a lot of sense for gateways and it also makes a lot of sense for meshes. And so we wanted to kind of lean into that. So if you're interested in kind of learning more, kind of diving deeper into a lot of these concepts around the way that Lincredi uses the Gateway API or other deep dive Lincredi concepts, there is a Service Mesh Academy that I can highly recommend. It is a monthly hands-on training. Their next one looks like it's from November 11th to 17th and if that is of interest to you, I highly recommend you check it out. You can check it out at buoyant.io slash SMA, Service Mesh Academy. And if you're interested in kind of running Lincredi without taking some of the administrative burden off, there's also a fully managed Lincredi which is called buoyant cloud and it can handle things like automated upgrades, version tracking, certificate rotation and a bunch of other administrative tasks that would otherwise fall to the clustered administrator. So if that's of interest, I highly recommend you check that out as well. And I also want to shout out once again, this is talk is by my colleague Matei. He did a really good job with it. If you liked this talk, please make sure you contact him on Slack or on Twitter and let him know. I can't take credit. But other than that, thanks for listening and happy to take any questions. You mentioned one of the cons is Ingress. Is it a replacement for your Ingress or can it complement your Ingress like a traffic? It complements your Ingress. So there's kind of, your Ingress may be using the Gateway API, but we are also using the Gateway API for East-West traffic inside the cluster. Awesome, thank you. Staying with the cons for a moment. So the other con he talked about was the Gateway API is kind of new, especially with working with a service mesh. So we don't have a service mesh yet. We have Ingresses that are con and I understand con can work with the Gateway API. So if I go back to my team and say, okay, let's start using the Gateway API for our con stuff and now we can start sort of getting into using Linkedee through Gateway API as well. Am I jumping the gun? Is this production ready? Or is it too early to be thinking that way? No, I don't think you're jumping the gun. So I think these are kind of mostly separate concerns, right, you've got con for your Ingress, you perhaps have service mesh needs, you can use Linkedee for your service mesh. I think there's gonna be a little bit of a confluence in the future between the types that they use, like if you have HTTP routes defined on your Ingress, you may be able to eventually use those to attach policy onto that Linkedee will respect. But there's not any kind of conflict between the two.


So just a couple of things, obviously with the injuries is some of the sub-final for Monday. We should have Elijah practicing, which is encouraging. Doesn't mean he'll play, but at least it's stepped in the right direction. Hopefully get his clock started coming back. Kyle had his procedure yesterday. He'll not see him the rest of this year. Nothing that concerns us going into the 23. But so that's kind of where we're out of that. TQ, have an update later this afternoon. See how his procedure goes. And then the one guy that was from last week that was questionable that will not be at practice today will be jailed and dulled. Everybody else will be back out of practice and we'll just have to see how the week goes. As far as Kyle goes, was the anticipation always that he was potentially going to be out for the year or did? Well, you know that he's going to be close. And he just needed to let the experts and you get the best in the world to get their advice and their opinion. And that was what we were waiting for. Until you examine it, then if you're going to have any kind of operation done, you don't know until they get in there. So that's why. Just to be clear, when you say off season, do you mean like the start of the TAs or do you mean like the more of the start of training camp? Except to see. But nothing that concerns us for real games in the season. Did you say what the procedure was? Like was it like an east opening or? I'll let him speak on that. Keep those guys' information private. They want to speak, post, tweet, do a tick talk about it on them. But give you a timeline and that's the best I can do. In terms of Jalen Mayfield, I know this is kind of the last one. It is. It is. I guess how much goes into play, his progress versus maybe what else you have at left guard? Because that's been a sport attack. It comes down to his progress and what's the best for his situation this season in his career. And you're just kind of assessing and appreciate him. Working back from the back injury, just kind of see where it is, his core strength and how effective he could be in a real NFL game. That's what we've got to be able to see. And with him, he's been working, it seems like, attack with a baby. Is he going to see that? That's our necessity. Because we had a couple of injuries last week. And obviously Jalen played guard force, played tackle in college. You need that versatility if you're not a starting lineman in this league. And if you're not starting right now, for sure. And then a lot of that might just add a necessity, just because the numbers you get down late in the year. And Chumo was doing something last week and just short numbers.  Again, I guess what I was asking is, do y'all still envision him mostly as a guard, or is he maybe? Yeah, and then we'll obviously see where our roster's at in the spring if something were to change next year. But for this season, it would be more interior. When you are kind of thinking about his progression, how much of it goes into the overall conditioning of a lineman and how different that is? It's not just like running around and getting conditioned, but having a different type of conditioning for a lineman. Yeah, I mean, there's a lot to that. When you've missed a significant amount of time, and yeah, I mean, there's those reps you can't replicate, no matter what kind of PT you're doing. And again, you've got to assess can you function through 65, 70 snaps in an NFL game? Hold up in protection, run blocking, all that stuff. So yeah, there's a lot that goes into it. And obviously, most important is his health, where he's at. What do you see in front of the rookie receiver, George Pickett, on the table? What challenges does he present? Yeah, talented player. Made a big play there tonight. Right there on the sideline. Picket through to him. Physical player, they got a good group of skill players. So it'll be a challenge. And certainly, he's somebody that you can see that Kenny Pickett's getting comfortable throwing to, and somebody you got to account for. You mentioned that he'd be on the Ritz-Kins team that I think he said went five consecutive to make a playoff push. Yeah, that's four. I was curious if you'd gone back and thought about that winning streak and anything you'd learned that you're trying to apply now to the end of this season to try to get your team into the post. Yeah, there's lessons learned on the flip side, whether you've been a part of things, if you come out of the gate strong and then there's a late season collapse, or in that situation, obviously, that was so unique dealing with tragedy and where football really seemed trivial at the time. But obviously, going back to that football analogy, where we were, and as these games get late in the season, I mean, they just came down all of them, came down pretty much to the end. Certainly the one where we had the first one you got to get, and then just kind of momentum carried from there. So there's a lot of things that lessons learned, but also just the reality with five games and you're kind of still in the middle of that path, or not really in the middle of that, but you've still got a chance at that perspective. So even if you feel like you're down two games, like the reality is if you can just, and nothing's more important than this one, you can get the first one. The unique situation to have this late of a buy, and you got a whole quarter left. So that's why they're all important, but the reality of where we're at situationally, and the opportunity available is unique in my opinion. Talk about how much the rookie, obviously, when you drafted this rookie class, knew how much of a role they were going to play pretty early in their rookie years, but a guy like Arnold Evercady, I know he's working through that arm thing, but how much does he kind of epitomize the growth of where's the guy you needed to start early and kind of where he is now? Yeah, I mean, he's taking a lot of valuable reps. He's made some key plays at times. Doing a decent job. I mean, he's got a pretty high percentage of pressure on the quarterback, unnecessarily. That's why sometimes sacks can be misleading. You'd love to have sacks, don't get me wrong, but just affecting the quarterback can move him off the spot. So he's gotten better every week, and we're happy he's here, but same thing with D'Angelo, more replays, you can see his progress, the things he does weekend and week out. When it comes to good outside linebackers, Ed Drescher, I mean, what is kind of a key marker of their progress that you see when you kind of, not just with AK and D'Angelo, but I guess just that position as a whole? Yeah, it just depends on the call, the scheme, because that job, if you're in base defense, as you're definitely an outside linebacker, you're really an edge player, whether you're going four down or five down, and can you hold the point? Obviously in the run game, most times that's what you're doing is you're trying to contain, you're gonna be matched up on open side tackles and tight ends, can you set the edge in the run game? And then obviously, depending on your plan and the pass rush, depending on the week, what you're trying to attack, can they operate in the rush? You know, they're a guy that just runs up the field, creates lanes for a lot of mobile quarterbacks, which you see happen a lot in this league, can they be disciplined? Can they know how to set people up and pick games? And then ultimately, can they win one-on-one when they get the straight rush? So those are all the things that go into it, the job description of it, and can they handle the call? So if you're dropping guys, and that's not the most normal thing they do, but you gotta be able to do it if you're gonna bring pressure and you're gonna play that kind of defense, and there's a lot black to it. And for those guys, there's a lot on their plate, and they both contributed in a positive way, and so hopefully we keep that going. You've played a lot of reviews, whether it's due to necessity or talent, for the last couple of years, and you've talked a lot about plans for players when you drop them out. How do you square, what goes into the plan that you maybe have versus the necessity or the need of playing a guy, because, hey, you know, you've got to do this. I mean, it happened to Mayfield last year, with Josh Andrews breaking his hand the Wednesday of week one. So yeah, so sometimes you've got a great plan, and you have to adapt, and so that's what you try to do, to have contingency plans, and sometimes it works out like I like Richie, and then other times, same thing with Troy right now, I'm turning in the right direction, but make it right. Sometimes guys get thrown into the fire, and that's all you got. So you try to manage that, they may try to protect them, certain calls, and certainly, at some point you're gonna have a hard matchup and they're having to try to protect them, but that's what you're trying to get them ready for. Make sure that they're contributing, trying to help you win, build their confidence, there is some psychology that goes into it well. Kind of going off that level, but what advantages or disadvantages is it for you game planning, going up against a rookie quarterback like Kenny Pickett? Well, I mean, he's played a good number of snaps, you know, obviously you saw Monday night when they were able to get the lead and keep it balanced. He's just a very smart player. You can see that growth, and there's a lot of things that happened, and I think the thing that goes on notice is how different these pockets are from college, that's the thing that you never can replicate, or damn near impossible replicate in practice. That's where the game's so different. The amount of time and the pressure players you have in this league, and how condensed those pockets are, and people at your feet all the time, how you adjust, it's easy to throw when nobody's at your feet. And so that's where the game is so different, in my personal opinion, from college to the NFL. Yeah, going back to Kyle for a second, now that he's officially done for the year, how do you assess what he did in year two? Yeah, I've been very different years, but going back to that, like I said, I mean, he had obviously bigger statistical numbers in the passing game a year ago. We were a much different team in a different place, where you saw a lot of growth with a lot of other jobs we asked him to do. Certainly there's some things that you wish would have gone different. I mean, that's probably every year, but you can see that growth as a player, as he's become a more complete player. And as I said, the best thing is Mike is, you're not wishing to open, you know he's done it, and you know he's made other gains in different areas of what we've asked him to do. So you're very confident and positive about where his future, predicting the future, and you can't predict everything, but you've got evidence, and he's made so much growth, and he's such an impactful player, whether he touches the ball or doesn't, and being able to run behind him this year, and some really efficient runs, and also too, and even on some of the actions where he's asked to block, where they're not just assuming that he's getting out, and I mean, it just makes it where, as he continues to build, and the cow is only 22. And you see this happen at other professional sports. I've given the example, you've seen the last two NBA and MVP's, people try to huge expectations, and they let them come along, and they become really dominant players, and at a really hard position in what we ask out of that position as well. And so we're very pleased with his progress. Is there, this might sound weird, I don't know, but do you ask more from your chat-ins than maybe some other schemes? Probably. I mean, the other thing we ask him to do. Yeah. I mean, I just, you know, sitting here and studying all 32 in the off season for you, but you know, you obviously watch a lot of different films and different things, and there's a lot of, so it makes this game interesting. It'd be boring if everybody was trying to do the same thing. Just generically, what does the Pittsburgh defense kind of challenges for your offense as you go there? Yeah, I mean, they've got some good veteran players on that front, you know, KMA words, that is good at careers. Anybody up there in a long list of, you know, right, since Chuck Noel got that thing going, and they've had some damn good defenders up there, and that's one of the highest praises I could probably give, whether it matters or not, coming from me, but that's why I got a lot of respect for Cam Hayward, what he does, anchoring in the interior. Obviously, TJ Watt, really good player. Tell, you know, coming back, playing through that pet injury, and still effective rusher, really good player, and then you got two pretty savvy inside linebackers, Devon Bush and Miles Jack, pretty familiar with his years in Jacksonville, and then that third level, I mean, because as good as anybody, it's a free safety in the league. Edmunds, those guys, they do a nice job. So, you know, it's, you see a lot of stuff that, you blink, and it was 15 years ago, whether it was Kiesel, and Casey Hampton, and Aaron Smith, and Lamar Woodley, and you get this group of guys now, there's a standard in Pittsburgh, and that's why I have so much respect for Mike Tomlin, and what he's done, and that guy I've looked up to this profession that's sustained a lot of success there. Anybody else? Any thoughts on my calls, my cleats, the guys wearing cleats, seeing the name? I mean, I've saw the list, I think it's awesome what they're doing. So, just one thing, if you know guys are passionate about it, and I think it's cool because it's on an individual basis, because you're never gonna have a group of people that, you know, everybody, it allows them things they're passionate about to bring awareness to, I think is a positive. I don't know. I don't think anybody's showing up to worry about what I'm worried. I had probably guys in his league that think that, that get all dolled up the sideline, but I'm just trying to coach football. So.


But I tell you, it's much easier to pick out our Sportsmax up moment. And you guessed it, it's the goal, Clement, one man show, and a top performance to put that into the back of the net. Juan Clement, wow, you can celebrate our Sportsmax up moment.


Hi, I'm Peter Burris and welcome once again to another CUBE conversation from our Palo Alto studios. Recently, we had Fordegard Labs here on the CUBE talking about a regular report that they do on the state of the security industry. And once again, we've got Anthony John Domenico here to talk about the most recent, the Q1 update. First of all, tell us a little bit about Fordegard Labs. Where does this come from? So Fordegard Labs actually is the threat intelligence organization of Fortinet. So what we do is we keep track of the tactics, techniques and procedures of the adversary and make sure that we have detection methodologies to be able to stop all those tactics, techniques and procedures. So you're the ones that are collecting the data that's right from the ground to help everybody keep up to date on where the threats are likely to be set priorities. So that's what this report does, right? Absolutely, it's something we do on a quarterly basis and it's really, you know, we're looking at billions of events that we're observing in real time, you know, production environments. And what we're trying to do is identify the top application exploits, malware and botnets. And what we want to be able to do is find different types of trends that then can be able to translate into helping organizations fortify their environments. All right, so here, this is the Q1 2018. People can get access to it. What's the top line change? Yeah, well at a high level, I think the, you know, one, the actual cyber criminals, they're evolving their attack methodologies to be able to increase their, you know, success rate, as well as being able to increase their infection rate. So that's one thing. You know, the other thing, obviously, we always have to talk about ransomware that, you know, seems to be a very hot threat these days for cyber criminals to make money. Now that threat isn't going away. We did see a slight decrease though where the adversaries were more interested in hijacking, you know, systems to be able to mine for cryptocurrencies that's opposed to, you know, taking that machine hostage and demanding a ransom. Really? Yeah, you believe it or not. I'm a little bit, I mean, ransomware just seems like it would have so much potential and cryptocurrencies are, well, they're interesting. But tell us a little bit about why that's happening. What seems to be the indicators? Yeah, well, like I said, ransomware isn't going away. I think they're going to continue to use that to make money. But from a cryptojacking perspective, we did see the uptake last year in our Q4 report it was about 13% of the organizations actually reported some type of cryptojacking attack. Fast forward to this report and it nearly doubled, actually over doubled to, you know, 28%. So that's about one in four organizations are actually impacted with this particular threat. Now, what I think is interesting about this particular threat is the way it evolves, right? Because it's so new, it's always looking back at his other successful, you know, predecessors to be able to determine how can I be more stealthy and how can I get my, you know, malware or my, you know, payload out to all the different sort of systems. So, you know, an example of that is fileless malware. Fileless malware is very stealthy. It's starting to use fileless malware techniques. It'll use scripts to inject their actual payload into memory, nothing on this so makes it a lot more difficult to be able to detect. Now, how do I get my payload out to all the other, you know, workstations? Well, it takes a one-two punch combination that, you know, Pechia used last year. It's leveraging, you know, there's this open source technology called, you know, Mimicats steals different types of credentials and does something called pass the hash, passes that hash credential out to those other systems and then it gains access that way, it can actually pass the actual malware from system to system. If that fails and then goes back to identifying different vulnerabilities that could then exploit, one vulnerability it does look for is eternal blue, which was the vulnerability that was so graciously given to us from shadow brokers. So those are the ways that they're starting to be more effective in being more stealthy and also being able to propagate a lot faster. And cryptocurrency obviously is one of the more interesting things because you take over the computer resources without necessarily stealing any data. You're just grabbing computer resources. Yeah, which is interesting. I don't want to actually kind of go off topic here but that's another conversation. Is cryptojacking actually a threat or not? Right, because all it's really doing is stealing, you know, CPU resources. So, you know, so people say, so that's a whole another discussion to actually get into it. Is it actually really a threat or not? Well, if you're able to get access to a computer presumably you're able to get access not just for that purpose but maybe others. Exactly. So that's probably an indication you may have a problem. But let's talk about ransomware. You said the ransomware is not going away. Ransomware most folks are familiar with it. What is it, what's the report suggest? No, Peter, did you realize that this month is the one year anniversary of WannaCry? I don't know if you remember that or not but, you know, WannaCry was very infamous for not necessarily the payload but by the way it actually was able to spread so fast and affect so many different machines. Now that spreading, that worm-like spreading kind of capability still exists here. You know, today you see a lot of different sort of threats using that but what seems to be a bit different now is the combination of that ransomware payload along with more targeted attacks.  Well, these are a lot more targeted so they're going to spend a lot more time doing, you know, reconnaissance on an organization and being able to find different vulnerabilities on the outside of the network once they actually come in, very methodical at how they're able to laterally move and put their actual malware on systems that they actually think, you know, however many systems they think they should actually have that particular malware on. Now at this point, they hadn't actually executed you know, the actual payload so they have it on as many systems as possible and once they're ready, they flip the switch and all those systems now are held hostage that impact is much greater to the business. Now, when we think about the attacks we think in terms of computing devices whether it's a mobile device or a PC device or servers or whatnot, but are we seeing any changes in how people are attacking other computing resources within a network, hitting routers and others to try to drive more control over some of these network resources? Well, I mean, we definitely see exploits that are actually hitting, you know, mobile devices they're hitting routers. A lot of IoT as well, but also web technology because, you know, web technology there's so much external facing websites these days you know, they're much easier target. So we are seeing that. I would mention also that it's up 7% to 21% of organizations have actually reported mobile malware as well. And that is a especially difficult thing because your mobile applications are not just associated with a particular business but other businesses as well. And so you are both an employee and a consumer and if your mobile applications get hit that can have enormous ramifications on a number of different levels. Yeah, absolutely. And I think sometimes, you know an organization or an actual consumer will have a phone and then when necessarily think that it's the same as their work station. So it's like, oh, well, not that much can happen on my mobile phone, right? They're not the same as on my work station but actually it could be even worse. Yeah, so if you think about some of the things that's on the horizon, you mentioned that we're seeing a greater utilization of different techniques to make money in some of the new domains like jacking, crypto jacking. There's still ransomware, still an issue. As folks go back and identify these different malware these different security breaches what are they doing to actually clean things up? Are we seeing folks actually cleaning up or is there still just like whack-a-mole just whacking things out and worrying about whether or not they go back and clean things up later? Well, to basically answer your question they are starting to actually kind of clean up but wait till you hear this. So what we tried to do here in this quarterly report is we wanted to measure how quickly that they were able to kind of clean up that particular threat and what we found out we used botnet alerts and we wanted to see how fast those botnet alerts actually got cleaned up. So what we were able to determine is 58% of all organizations within 24 hours we're able to clean up that particular botnet infection which is actually pretty good but that 42% it took them either sort of two days or longer to be able to get that actual threat out actually sometimes the threat really never even actually went away. A great example of that is actually the Andromeda botnet is a threat that was brought down last year but even though it's not there anymore the infections on the workstations are still there so we're still kind of getting those actual hits on that Andromeda botnet and that actual threat for Q1 was one of the highest in prevalence and volume. Even if it wasn't necessarily doing damage because we'd figured out how to deal with it but if it's there somebody might find a way to use it again in the future. Absolutely. So as we think about the next quarter you doing this on every quarter are there any particular areas that you think folks have to, they need to anticipate some of these changes more of the same different trends or what about OT for example as operational technology becomes increasingly part of that common technology fabric how is that likely to be affected by some of these different attack types? And as your first question I think we'll probably see a lot more of the same and I think what we'll continue to see this is whole zero day market I think it's getting more and more mature meaning that we're going to see more and more vulnerabilities that are actually kind of zero day that have just been discovered or just been announced and I think we're going to continue to see the adversaries take advantage of those newly discovered zero day vulnerabilities they'll take those actual those exploits put them into their attack methodologies to propagate faster and faster so I think organizations are going to have to make sure that they can address some of those newly discovered vulnerabilities fairly quickly. Now as we switch the OT side we didn't see a lot of attacks if you look at the percentage of the overall attacks however OT if there is an actual successful attack I think it's worth saying that it's a much larger impact than a major problem. My concern is these different types of trends that are coming together one OT is starting to connect to other networks which means they're going to eventually be accessible from the internet which makes it a lot more difficult to be able to protect at the same time we're seeing nation states continue to focus on compromising OT systems as well so I don't know what's going to happen in the coming months and years but the trends aren't actually looking so good right now. So if you were to if we had a CIO sitting here right now and you were talking to them about this report what are the first off how should they regard the information? What should they be doing differently as a result of the information the reports reviewing? Yeah I mean I would say one we always talk about this it's easier said than done but going back to the basics and making sure that you have good cyber hygiene and being able to identify vulnerabilities that exist in your environment and that me just saying that sounds kind of simple but that really means identifying all the assets that you have in your environment that you're responsible for protecting number one and then being able to identify the vulnerabilities that may exist on those things that's not the easiest thing to do but I think it's something that really should be focused on at the same time though threats are going to get into your network that's just a you know that's a given so being able to make sure that you can identify you know threats within your environment it's extremely important and then once you identify them what's the processes for you to go ahead and actually respond and clean up those particular threats that really is going to be the key I know it's at a high level it's much deeper than that but I think that's where you start. All right Anthony Giannomenico, Tony G. Tony G. Thanks very much once again for being on theCUBE and talking to us about four to guards Q1 2018 report from Fortinet. Awesome, well thanks for having me. You betcha, so Anthony Giannomenico a senior strategist researcher at Fort Agard Labs of Fortinet talking to us about the one Q 2018 report. Once again this has been a CUBE conversation thanks for listening.