Welcome back everyone to theCUBE's live coverage here. In Las Vegas, I'm John Furrier, your host of theCUBE with Dave Vellante, my co-host, co-founder. Heads up, CUBE Research. Got two great guests here. Gavin Day, who's the executive vice president at SAS, and Chris Devise, general manager, America's Technology, leadership team and platforms, and ISVs at Intel, which makes all the great compute that we need. Chris, great to see you. Gavin, thanks for coming back on. Thank you. Great to be here, guys. Appreciate it. So, obviously, Brian and your CTO came on. We just chatted.  It's like Star Trek, Scotty, more power. I need more power from engineering. So, compute is going to drive all the data analysis. We've seen that from, by the way, from quantum on the high end and research down to CPUs, XPUs, GPUs, all kinds of different versions of compute are coming. Talk about your relationship with Intel. Why is it so important for SAS? Yeah, I mean, this relationship's been 25 years in the making for us, right? And it starts with co-engineering. It starts with joint customers. It starts with joint go-to-market. And then from there, you mentioned that there's an absolute need for compute, but there's absolutely a need for the performance of it. And then also, how are we going to help our customers? Because the cloud spend is running out the door right now, right? And now we're partnering with people like Intel to get efficiency and performance. And it's been a great ride. Chris, Intel's built on partnerships. Your entire business is about getting that technology in the hands of partners. Yeah, absolutely. So that just sums up our relationship with SAS. We have, some of our biggest customers are common, and it's so much better when they can see the TCO opportunity of both of us working together because they know it's co-engineered, and they know how to co-implement it and get the best and easiest result for the best TCO for their use. You're one of the few companies that was founded before SAS. And you guys both were founded in the mainframe era, so okay, X86 didn't exist when you guys were founded. And of course, so help us understand the roots of the partnership. So it goes back 25 years. Take us back, and where did it start, and where's it gone? Yeah, so I mean, I can tell you from our perspective that our CEO is, Jim Goodnight, is very interested in the performance of compute and always has been, and had relationships with executives at Intel, and through that, started talking about the needs that SAS had from a compute and a design perspective. And then that led into, like I said, joint R&D, so designers from their side, engineers from our side, working together so that way we can bring the best solution forward. And one stat that's pretty cool there is over this period of time, we're looking at upwards of 90% of our customers are running SAS on Intel chips. So they're a huge market leader there for us. What are some of the co-engineering things that you guys have done? Can you share a little bit about some of the benefits that come from that partnership on the SAS level? I mean, first of all, it comes with, we're designing these offerings for our customers. And then when we look at optimizing SAS specifically to run on Intel chips, we don't do that anymore, right? And then we take that out a little bit further and optimize with our cloud partners to run our software in AWS or Azure on optimized Intel chips. And there's, you're obviously servicing a lot of highly regulated industries. And I'm interested in if there are any things beyond, anything you're doing specifically around security and if not, are there any security features that are advantageous for the SAS customers? Yeah, one of the capabilities we've worked on together is trusted domain extensions of TDX. It's, just think of it simply, it's like putting a moat around your castle. So it's that extra layer of security in the hardware and it's SAS Viya is completely validated and optimized to use this capability where it's available in the cloud or on-prem. You know, one of the things that we always go to these events and stuff, we're in a market transition. Obviously, Jen and I, everyone's seen that. And the GPUs, everyone is hoarding them. They're trying to bow grab as many as they can. But in all the old days, I remember old days when I was 10 years ago, maybe just go back 10 years, the word one-off was a bad word. That's a one-off, we don't want to do a one-off. One-off meant it wasn't optimized or purpose-built even was like, purpose-built better be good. Now we're in a world where everything's a one-off in generative AI. Technically it's like these things are being built out, new things are being generated, so that's a phenomenon. And then you have existing workloads that were existing that were known and deterministic before generative AI. So you now have this collision of deterministic workloads with now non-deterministic features. So does that change the paradigm on how you handle where to put the workloads and what compute to assign to it, how do you think about that? Yeah, I think some of the SAS Viya features that have just been announced, you can use all the standard great SAS features that have been around forever. And now generative AI has become the big workload everybody wants to look at. So we've worked together to fully optimize around technology we have in our chips, in our Xeon chips called Advanced Matrix Extensions. And it basically is a matrix multiplier. So it's got a big cash in front of a matrix multiplier function for each core. So think of it as your gen AI accelerator right next to each core to accelerate it, just like SAS is offering your traditional predictive analytics type of workloads using statistics right next to generative AI. Each of our cores has that capability in it. So you get the best power, the best TCO, the best efficiency, but what it really means for the Fianne user is like, I don't care where my workload or where my data is, if it's running on the Intel chips, I could take advantage of it and work smoothly and easily. Okay, please. The workload and where it runs is an interesting conversation right now because early days cloud lift and shift was the model, pick it up off-prem, moving into cloud, all of a sudden that's not performance, costing customers too much, and now we're having very real conversations with our customers about where they're running what workloads and why. And I think part of that is absolutely due to the cloud spend that we're all seeing, right? I see that bill every month here at SAS and I know what we spend across our crowd providers and so CIOs want to be thoughtful now, right? And it's a huge part of our message to be able to go to them and say, this was engineered jointly between SAS and Intel and these are the benefits that you're getting from it. I'm smiling because Dave and I tend to take on topics that are kind of cutting edge in the front lines, but also boring topics that no one's talking about that's super important, like governance and TCO, right? You mentioned TCO multiple times. These are two very important concepts that are really front and center because costs are everything if you don't have a TCO model that works, and by the way, half the workloads right now we're seeing and reporting on our research don't need the high end gear, right? So can you guys talk about the TCO equation and then there's a mix and match factor. If you need to throttle something up super high and you go with something there but the general market, that's a bunch of one offs generative needs general purpose computing because they're compute hungry for the SAS application. You guys crunch data, right? So this generally, I don't know the general purpose that's made me a bad word, but like there's a market and cost matters. Yeah, one of the things that we're working on is having SAS Viya be smart enough to guide our customers on where they should run certain workloads because of the either economic or time impact and as we get better with that with our customers we're seeing the performance and TCO go down. TCO is important to us because SAS will use all the compute you want to give us. So we have customers that are saying, hey, I'm spending this much on SAS and potentially a fraction of it is the licensing cost. The rest is all the power under the hood. So TCO is top of mind. What are some of the TCO variables? They're probably the same, energy's probably a big part of what's the TCO calculation look like? Because they're going to eat all the compute they can get and they're like always hungry, appetite's huge. Yeah, so power, time, how long it takes to run the model? So that's where performance comes in. And then when you start looking at generative AI models, you need a different set of hardware to train a multimodal model, a large parameter multimodal model. Everybody thinks a chat GPT, you need dedicated GPU accelerators to do that or AI accelerators. But for most enterprises running AI workloads if you're doing generative AI, the compute requirements become much smaller. So if you're, and a lot of people are talking small language models or using RAG or all these other techniques to take their data and use their own data, their own IP, make sure it stays in the confines of their walls and not to go train some other model that everyone gets to use, they could use that as very efficient and they get the results quickly. So low spend, high performance, great results, easy for the customer to implement. You mentioned Xeon before, is that really where the focus of the partnership is? Because of course there's a lot of discussion in the industry about CPUs and GPUs and NPUs and ColoXPUs, all the silicon diversity. Is it the case that the general purpose computing paradigm works well in your world or are those alternative silicon designs coming into play? So the 25 year history Gavin's talking about has been largely CPUs, because that's been our biggest business, that's been our biggest relationship together. And as we've enhanced the CPUs, we've done more and it's been from the compiler level up. Very, very strong deep technical relationship. Well, we've just introduced discrete accelerators. We just announced the Gaudi3 accelerator. We're working with SAS to look at how do we integrate that with our products? So there's alternatives for TCO, for the customers who need that additional compute power, if they're going to be training large multimodal models. I like the hook example, where you just talked about the Viya figuring out where to get the intelligence to place the workload. Because I think we're going to have a distributed compute environment as well. So what we're watching right now is you can have a spectrum of portfolio of power, if you will, compute power, not power power. When you say, okay, I got this over here, I got this different configuration over in this side of the cluster, I'll run that over there. So maybe reasoning goes over here. Or I want prompt response here, IO basically. And the user doesn't care, right? It shouldn't care, right? The economics of it and CIO cares, right? We want that, and that translates from compute to storage, because one of the, I'll say errors we saw early in the, as the cloud expanded was, I need a whole bunch of CPU power, so I'm going to put this on bigger machines. And it was a whole bunch of IO and a whole bunch of memory that the customer was spending money on that they didn't need. So that part is absolutely important for us. And what's the customer, your customers can figure out what they look like with Intel? Is it generally like, okay, if you're running X processors on Intel, Xeon, are all the, what processors are you guys supporting? And do they buy clustered systems? Are they racking and stacking? What's your customer environment look like? It's all of the above, right? We have deep partnerships not only with Intel, but with Dell and a whole bunch of other companies like that, so we're going where our customers are. All of them know how to size them and do the architecture for SaaS. We've done that work over decades, or as we go into optimized compute within the machine types within Azure. So one of the things you've heard this week is, we believe we have to be able to say yes to where our customers want to run, how they want to run, the languages that they want to use, et cetera. And all generations of Intel, right? Absolutely. So it helps get the most other existing investment. And if they choose to go to a new investment of newer hardware, they'll get the absolute most out of that new investment.  Okay, it was Interconnect. That's Ethernet. Yep, I mean. Sounds like it's Jada Center to me. Exactly right. I've been in this industry a while, and from an infrastructure standpoint, customers want three things. They want it to be rock solid, they want it to be lightning fast, and they want it to be dirt cheap. So run your software on that, and I'm happy. That's absolutely right. Yeah, let me do my stuff. No one says I want a slower processor. Yeah. It says that. And slow and more expensive. No one's calling me without it. And that's the TCO equation. It's like nobody wants to go backwards. I'm telling you, the stuff that's sexy is TCO and governance. Those are the two factors right now that are the hottest things that we talk about. Everything comes back down to, if you don't get the governance right on your AI, you're pretty much screwed downstream. And then obviously on the performance side, you got to have the right cost structure because the cost envelope's driven by power constraints. And also, why am I spending all this money for this gear I don't need? Yeah. The workload doesn't need it. Yeah, absolutely. And I think just to add on to that whole governance part we were talking about, SAS has uniquely put all the models together in one place so you can make sure your data is enhancing your business. And you could use it a way that's easy to improve your business. So you could have traditional models, new gen AI right next to it. And then underneath that, whatever you're the hardware is you're running, you're going to get the most out of it because of our long history of working together. Chris, that is a great point. I want to double-click on that today. We were talking about earlier that is exactly why latency is huge in models. So if your data can't get delivered to the compute, it's not factored in, then the whole model could crash. It's all about data availability, right? So like, I mean, this is just basic networking concepts. Move this point A to point B. It's stored here and that's got to move over there to be served. Yeah. If you don't hit the latency number, your AI doesn't really work. Yeah. And there's the performance and the chip side of it with Intel, but it's also, we know data has gravity and we're trying to talk to our customers about not moving data around nearly as much as they have either on-prem or within the cloud because there's your cost and performance issue. If your governance architecture is wrong or the overhead of that is huge, it crashes the value of AI. Absolutely right. So Raj was saying on the stage this morning about moving the data. You don't have to move the data. Single store, right? That's his whole wrap. We're taking a premise on our research right now. We haven't published anything yet, but we're starting to take the position that you should be moving data around a lot faster and that we think there's ways that do new architect and switches in the new configuration where you can reward from good new networking techniques massive data movement. That doesn't cost an arm and a leg. I like the derivative of the Einstein quote. It's not his scope, but I apply it. Move as much data as you need to, but no more. No, there's truth to that because you run into the laws of physics and speed of light problem and there really is a cost. There's a power cost to moving the data around. So depending looking at your compute setup on where the data is and data gravity does become important. So if you want to do it at the edge, how much compute do you need at the edge? If you want to do it at the near edge, if you want to do it at the data center, all those factors play in a place and Sass had a great case study on stage around Georgia Pacific. I went through each of those examples of how they do compute and why it needs, that data needed to be stay local in some of those cases so the data, so there was real-time access to that data. That was a home run on day one, if you haven't seen that, check that out. But speaking of physics, did you hear Pat sort of brought forth his three laws? Pat Gelsinger. Pat Gelsinger, we first heard this years ago on theCUBE when he was the CEO of VMware, right? The laws of physics, the laws of economics and the law of the land. He's now applying that to Intel's business. He's recycling that, but it's good. Is there all three of them? Immutable. His other line is if you're not out on the right wave, you're going to be driftwood. You're too far out front. You driftwood. You don't want to miss the wave but if you get too far out front, you're driftwood. Yeah, you're absolutely right. You have timings, everything. Welcome to this business, right? So what's next for the partnership? Obviously we're in your hungry for compute landscapes and distributed computing. Architects are now seems to be the AI thing. That's what's been validated. What's next for the partnership for you guys? Yeah, I see us continue to work with Intel technology. So more of what we've been doing, latest, greatest processors supported. But as we've talked about GAUTI3, is we're very interested in working together under that as SAS enhances and expands its capabilities on SAS Viya. We would like to do the same with our hardware portfolio and our GAUTI3 was rolled out. One and a half times performance of the current best product out in the market. 1.4 times more power efficient. So a lot of opportunity for SAS customers to take advantage of that and have some choice of their options no matter what they're doing with their data. And that's what we want to provide ease of use, TCO and choice. Yeah, your comment on Driftwood is really true, right? One of the things that we do with all of our partnerships is make sure we can center a customer in the middle. So do we have a customer that's bringing us a problem that we can go solve with Intel and the partnership's going to continue to be anchored there because we don't want to solve problems that no one's asking us to. We need to solve the things that are keeping our customers awake. Awesome. Final question, just to wrap up. First of all, thanks for doing that. Final question, what is the most important story in this AI world? From your beginning where your practitioner had or industry had, you don't have the way the company had if you don't want to. It's too controversial, but what is the most important story in this AI conversation that people aren't talking about that should be talking about? I think from our perspective, it is absolutely our customers and the market being measured. And what I mean by that is I believe we are getting ready to have the summer of disappointment on AI. People are investing large sums of money, right? They're doing a bunch of awesome science projects, but is it bringing them the value of the investment that they're getting? And we talked about it earlier this week of being very measured on stage, making sure we're using generative AI, where it makes sense and where it helps creativity, but not trying to always solve the every problem that we see with AI. So overhyped is pretty much overhyped. Everyone's talking about the hype, not the meat and potatoes, chopping wood, carrying water. I think a lot of the hype is warranted, right? It is going to, this is a transformative technology, right?  And we don't have to try to solve every problem with an AI model or a generative AI model. Let's use it where it helps. That's exactly it. I'm right there with you Gavin, is getting the TCO right. And it's looking at, and I love the French rugby team example that you had today, not only because I played rugby in college, but also because they used all the traditional analytics, and then the generative AI right next to it to augment how you're thinking about the analytics. And when I've talked to the best uses of AI today of people getting results, it's that kind of assist you to help the other tools and how you're thinking about it. For example, like, hey, everybody thinks it's going to replace writing code. Most software developers are spending, they're spending a little bit of their time writing code. The majority of the time, they're spending like architecting or thinking about what's going to happen. And nobody likes writing comments. So it offloads some of those to make them better at their core job at the whole software architecture. So in this case with the French rugby team, it showed them how to use all these other tools they have, traditional tools, it made their team better with the generative AI working together. So I think it's the, how do I use it? And it was done in a cost-effective way because it was done with their existing tools. Yeah, productivity and good ROI on that. Yeah, absolutely. Yeah, awesome. Guys, thanks so much for coming on. Thanks for coming on. Chris, appreciate your time in Intel and Gretz. We love what you do. We mean more power as they said in Star Trek. Captain Kirk Holm, that's got any more power. We're here for you. The crystals aren't working. Thank you so much. Thanks guys. Thank you guys. Thank you. All right, we'll be right back with more coverage. I'm John Furrier with theCUBE with Dave Vellante. We'll be right back.


I'm very happy to be here today. I'm Kendall Nelson, like they said, from the Open Infrastructure Foundation. Joining me today, we have Alex Song, a contributor from Cyborg. Alex, maybe you can tell everybody a little bit about what Cyborg even is before we jump in. Yes, I will introduce Cyborg Projects. Cyborg is accelerator management projects with management, GPU, IPGA, SSD, and so on, as well as hardware, and software such as DPDK and CPDK. Cyborg Project has three micro-service, several APIs, several conductors, and several agents. Cyborg API provides API for resources, such as devices, deployables, device profiles, and so on. Cyborg Conductor connected to the device to, and Cyborg agent runs on computer node, which contains accelerator drivers, such as GPU, VGPU, driver, IPDA, driver, and the MSSD driver, and so on. We can set the drivers on computer first, and Cyborg agent periodically finds the devices and gets the device info. Then sends this info to Cyborg Conductor. Cyborg Conductor compares this info with the old devices, and updates these two devices, and this Cyborg can communicate with other components, such as NOVA, Neutron, Glance, and Sender to use these devices. Awesome. Thank you so much for that summary. I know Cyborg is a very bigger, up-and-coming project and certainly complicated in how much it has to work with NOVA and Neutron to focus on that hardware acceleration service. So I know that there's been a lot of work done between Cyborg and NOVA, particularly in the Xena release. Can you talk a little bit about the hardware enablement that as far as what happened there, this last release, and what also is coming up in the yoga release? Oh, yes. Cyborg products will have some interference and operation optimus and many drivers to support, such as silence, the IPDA, and internal pre-processed memory. And the enhanced about the accelerator, accelerator instance to migrate and resize. And we help team is the API filter, API filter conditions and the accelerator bound process, and so on. Thank you. Okay. A lot of work being done in the drivers and more focus on hardware acceleration, which is what Cyborg is all about. I know some of the work that went into the hardware enablement was around SmartNICS, and I hear that you have a demo to talk through for us today. Can you explain what is going on in that demo? Yes. Firstly, from the demo we can see that firstly, we need to create a device profile. The device profile is many kinds of accelerators combination, such as SmartNIC and NGPU. Then the NOAA API puts the instance with device profile. NOAA conductor calls NOAA scheduler to select available accelerator host. NOAA scheduler called placement to get the allocated candidates. Then NOAA conductor calls NOAA computer to put the instance. NOAA computer gets the instance and accelerator releases from Cyborg API and so it's this info to its XML file. Then NOAA, the instance can envisage visits as hardware from the host. Yeah, it's a lot of effort has gone into getting this to land between NOAA and Cyborg in particular, but also a little bit of work on the neutron side. So it's very cool to see all of this come together and as a part of the Xena release. So we're so pleased to be able to share what's been going on in Cyborg in both Xena and yoga and we really appreciate you all being here today.


Welcome to Checking the Chains at the Gate. We're going to talk about how we can build supply chain security policies using Gatekeeper and a project called Ratify. My name is Jeremy. This is probably my favorite contribution to Kubernetes. It's my cat with a laser background. He was the 120 release mascot. I was the 120 release lead. So that was the fun perk you get for being a release lead. You get to come up with a fun mascot. I'm also a chair in SIG Release, and I'm also on the Kubernetes Code Conduct Committee. So you may know me from one of those things. In my day job, I am an engineer on the Azure Container Upstream Team, and we are really focused on making sure that open source and Microsoft have a good time together, specifically around containers. We're looking at things in the supply chain security space and making sure that we are doing good things with open source and helping open source be successful as well. Before we get going, let's kind of take a moment to pause and reflect on the state of building and shipping services and software today. It's pretty good. There's a lot of tools that we can use, a lot of languages, a lot of frameworks that we can take dependencies on to just satisfy the needs and allow us to focus on business use cases. We don't have to go and reinvent the wheel all the time. Sometimes we do, but not all the time. And generally, it just makes things faster. The downside to that is that there's a lot of things. There's a lot of complexity to worry about. Building services is easier than it probably has been in a long time, but operating those services and knowing what's running in different places is a little more complicated than it used to be. As we start building microservices and shipping things to more clusters and different clouds, it's really difficult to kind of understand, what am I doing in Cloud A and Cloud B? What's this U I've got between those things? How do I keep track? How do I enforce that we're doing things consistently across those things? And to make matters worse, a lot of governments are helping us by making new regulations and directives that we have to comply with. The EU, this is an example, has mandated that people start to produce S-bombs. The US government has done the same thing. We at Microsoft have had to react and start to build S-bombs for Windows. So think about the complexity that comes into that. And all of that happens on top of the existing complexities and the difficulties we have with everything that we already have to deal with. And to make this problem better, we've added new tools. So instead of just that one S-bomb thing, maybe we've got two tools or we've got a series of frameworks that are coming to help us take that requirement and turn it into reality because things don't magically appear by themselves. How do we take all that stuff and actually make it useful? If we're just building an S-bomb and storing it somewhere, does it really satisfy that requirement we saw earlier? That requirement was we need to produce S-bombs. We need to produce bills of materials so that customers can have better visibility into what we're building. They can make decisions. They can have a better understanding of what vulnerabilities might be in that software. And if we're just producing this thing and then maybe sticking it on a website or sticking it in a SharePoint site or a ShareDrive, is that useful? I think for these things to become the tool that they're intended to be, we really need to start looking at automation and figuring out how we can make automated decisions about these things. Maybe the S-bomb declares that there is something bad in it, and we no longer want to use that. We need to be able to do that in an automated way, not with a Excel spreadsheet that has 7,000 tabs for all the clusters that we're running. So with that in mind, how do we enforce things in clusters? How do we define policies and control what we're going to do? So we're at KubeCon. Let's scope this down to what we can do within our Kubernetes clusters themselves.  But we can control what's in our clusters. So what do we do? How do we approach this? If you're using Kubernetes 126 or later, like the cell admission policies might be a really good way for you to start to build policies and apply those to your clusters. You can declaratively make policies and to say, I want this pod to have certain labels or things can only come from certain registries. It's really useful to declaratively say how you want things to look coming into the cluster. But those things are limited because they're running in process. You're limited in what you're able to do. You can't reach out to external data sources. You can't do really complicated things with them. So an answer to that might be to use admission controllers. And admission controllers have been around for quite a while. If you haven't heard the term before, what they basically allow you to do is to hook into the flow. So you type kubectl apply or kubectl create for a resource. It's going to go through a flow. And controllers can plug into two spots within that and they can change the request. So you can mutate your pod definition maybe to rewrite to a different registry or to set a default number of replicas or add labels, maybe billing information for a team that's going to deploy. And then you could also validate those requests. So as it comes in, you can say, you're deploying out of a GCR registry. We don't want to use public internet. Let's block that. So it gives you that kind of hook to be able to do that. But the downside is that you now have to write or run some component in your cluster as well that is just another piece of complexity. All these things, we're going to work on evolving to satisfy these requirements. But there's going to be some trade-offs. So are there pieces of software that we can use to plug into those spots?  There's a bunch of really great projects in the ecosystem. But we're going to focus on OPA gatekeeper and start from there. There are alternatives. You can take a look at Kiverno or Sigstor's policy controller. But OPA, I think, gatekeeper is a really great project. And it has a cool story that we're going to see unfold through this. So how does gatekeeper work in that scenario? So gatekeeper is based off of Open Policy Agent. And Open Policy Agent uses a declarative language called Rego. So you can write policies in Rego and then have that applied in either of those phases of the emission flow into this cluster. And that works by introducing a couple of new concepts. So we have constraint templates and constraints. And you can think of these as new resources that gatekeeper is going to add into your cluster that allow you to define those policies and then apply those policies to certain sets of resources. So let's take a look at a constraint template first. The constraint template allows us to define parameters, what things we want to include in our policy, what data do we want to use as configuration. So we've got a schema here that defines a parameters field with a label underneath of that that's going to be an array of strings. So in this case, we're going to look at how we can apply required labels or required labels on things coming into the cluster. So our parameter is probably going to be an array of those required labels. I think that is pretty straightforward. Once we have that, then we are going to define the actual policy itself. So in this case, we're going to take that object that's coming in that's under review. We're going to get all of the labels out of it. So in a Kubernetes resource object, there's always a metadata section. Metadata section has labels. So you can write a pretty standard line here that says, in that object, give me all of the labels from the metadata section. Once you have that, you can get that parameter that we defined, which is going to be those required labels. And you can compute the difference. So if we take the required and the provided and there's any difference, then we can generate a status message or an error that says, you did not provide all of the labels that are required. And then Gatekeeper can evaluate that policy and say, I don't want to admit this into the cluster because it's going to evaluate that. So once you've defined the template, then you can define an actual constraint. So we want to apply this to namespaces. So our use case here is all the namespaces in our cluster have to have a set of required labels. In this case, it's just going to be Gatekeeper, simple. But it's going to match anything that's coming in that's creating any namespace. OK, so let's see what that looks like in action. So we're going to, oops, did not play. It was? Oh, OK, there it goes, cool. It's not showing up on my speaker view, that's why. OK, so we have that constraints template and the constraint itself defined. We're going to apply those into the cluster next. So we're going to use kubectl to do that. And that'll be in just a second. So once we've applied that, can we make a namespace? That's kind of the question that we're going to answer. So we're going to apply those things. We're going to do our kubectl apply with the constraint template and the constraint. That's going to configure Gatekeeper. Gatekeeper's already been loaded into the cluster behind the scenes. Not super useful. Now we can use kubectl apply to try to create that namespace itself, and we can see we get an error back that we don't have the required labels. We have to provide the label Gatekeeper. So that's pretty nice, right? We can apply that policy, and it's good to go. But as we're starting to think about policies, especially in the supply chain space, I think things are going to change over time. If you've ever run a vulnerability scanner on something, today is going to be different than what you get tomorrow. So a useful feature might be to not just look at things that are coming into the system, but maybe audit things that are in the cluster already. That's something you're not going to be able to do with something that doesn't know how to do auditing, but Gatekeeper luckily does. So Gatekeeper allows us to do auditing. So we can apply our policies to the cluster, and if auditing is enabled, then it's going to go through on a periodic basis and look and see how do the things that are in the cluster now line up with what's defined in the policy. So we'll get an audit timestamp. We can see when the last audit happened and kind of use that in two ways. One, we can look and see, do we need to make tweaks to our policies and will things get blocked that we don't want to get blocked? But also it'll give us a good awareness of how things have drifted from what that policy might look like. So in the case of our namespaces, this is my kind cluster, and I applied that policy after the kind cluster came up. And obviously, kind is not going to add the Gatekeeper label to all of the namespaces that I have in the cluster. So here we can see that all of those existing namespaces now would violate that policy and would either need to be remediated or we'd need to tweak the policy to exclude a few things. So that's pretty cool, I think. I think you can get to this point where you're labeling things automatically, you're enforcing that policy that you have decided makes sense for you. But that's really working on things that are in the cluster. And I think for our supply chain use cases, that's going to be a little hard to do. Are we going to take all of our S-bombs and load them into the cluster as some custom resource? That's probably not going to happen. So how do we deal with things that are not in the cluster themselves? And what kind of use cases do we think might be necessary there? So maybe we want to validate the integrity of images. We want to make sure that the image that we pushed as part of our release pipeline is what we thought it was at the time that it was published. So maybe we want to do verification of the images. Maybe we want to take those S-bombs and block deployments based off of their contents. Maybe we know that Log4j was used back in 2021 and it had a bad version. How can we use an S-bomb to remediate that situation? So maybe we want to do that. And then finally, maybe we want to leverage vulnerability scans. Maybe we have a nightly process that goes through and scans all of the images that we are using in production and attaches a Sarah report out of trivia or something that has all the vulnerabilities that have been identified. And then we want to block anything or maybe audit everything that has critical vulnerabilities that align with our compliance requirements. So to do that, we're obviously going to need some pieces of external data. Those aren't going to be defined in the pod spec. They're not going to be part of that core Kubernetes resource that's going to be part of the admission flow. So to do that, we already know we're using Gatekeeper. And luckily, Gatekeeper has this really cool feature called external data. So your scenario is probably like this. You have a cluster. Gatekeeper is running inside of it. And an admission request comes in. In that admission request, there's some data that you can make decisions on. But now we want to go and do those external cool supply chain security workflows. So what we need to do is figure out how to plug that into Gatekeeper. And Gatekeeper does that with external data. And you do that by creating and registering providers in your cluster. So provider is going to be that specific piece of functionality that knows how to go and do something else aside from what's in that request. It might query a service or it might pull some data down from somewhere else, from your secure data location source that's off to the side. So what will happen is when that request comes in, you've defined a policy that says, use external data to get an answer based off of the data that I'm going to send to you. So the flow looks kind of like that. Your request will go out to the provider. And then the provider is going to talk to your external data source and then say, yes, this is good or no, this is bad. It's going to make that decision and return that back to Gatekeeper so they can evaluate the rest of the policy and figure out what needs to happen from there. And that looks pretty similar to any other policy. You just need to specify that you're going to use external data, which provider you want to use. So you can have multiple of these things set up and then what you want to send to it. So we're going to, in this case, grab all of the containers and images in the containers from this template. So this is probably a deployment or something that has a deployment template inside of it. Once it gets all those images back, it's going to send that to the external data provider. The external data provider will do the thing that it needs to do and return you back an answer or response. And then you can take that response and figure out, was this successful? Was it a failure? What do I do in the next step? And then code all of that into your policy. OK, so now we know we're going to use Gatekeeper. We know that external data allows us to reach out and touch some other service. This may be in the cluster. But how do we get those things? How do we distribute those S-bombs? How do we distribute the signatures, the vulnerability reports, or anything else? This is a space that's going to keep emerging and changing. And new things are going to unfold all the time. So I think one way would be to put those into file service. You could maybe put them in blob storage in your cloud provider and query things and find them. That's a fine way. But how do you associate those things with the images that are coming in?  It's got two containers. Those two containers have S-bombs associated with them or signatures. How do you know which one is which? Well, in the new version of OCI, 1.1, which is going to be approved hopefully soon, there's a new API called Refers. And Refers allows you to say, Object A, Object B, have a relationship, and then you can kind of query that directed graph. And a fun fact is that you can use OCI registries to store other things. So it's not just images. If you're using Helm, maybe you've pulled Helm charts from an OCI registry. Flux does that as well. You can store Flux things and pull them from OCI registries. If you use SigStore, they are now storing signatures in registries. When you co-sign something, the signature ends up in the registry. And you can find it that way. The Refers API takes that to the next level so you can store all kinds of things and associate them together. So you get this nice graph of, say, Object, maybe my container, Sbomb, and maybe a signature. And the way that that ends up working is that you get, hopefully, in the registry near you soon, you get your image. So we've made this image index that has an ANB64 image for our internal application. And we then create a new artifact, just another thing that we're going to store in the registry for our Sbomb. There we go. So we use an image manifest to create that new Sbomb. The really, really critical thing for this workflow to work, though, is that subject block that's at the bottom. That subject block is going to refer back to the thing that it's describing or that it's related to. So if it's a signature, that signature is going to refer back to the container. It could be a signature that refers back to an Sbomb. The really cool thing that you can do here is that you can have that graph walk down several levels. And we'll see that in just a second. It's going to refer back to that. And then the Refers API will allow you to do those queries and say, for this given image, which is shot2564413, oh, sorry, shot256892053, et cetera, give me all of the objects that refer to that. And it'll give you back that whole collection. And you can filter it down later. And you can say, give me all of the Sbombs that refer to that and pull those down as well. So we know that OCI is going to be useful here. Let's take a look at maybe a notional application, how this might work for folks. And this is going to be a really simple one single container application. But we've decided that we are going to sign it. And we're going to produce an Sbomb. And we want to publish that Sbomb. So it's going to be that kind of complete workflow. And to do that, we've enabled our teams through a reusable GitHub workflow that installs all the tools and then signs with Cosign. It also signs with a new tool called Notary or Notation that allows us to do slightly different things. And then we're going to generate an Sbomb and attach that to the thing as well. So it should end up with an Sbomb and two signatures. And what that'll look like in the registry once we've published that is a little something like this. OK, there it goes. It's playing. Cool. So we're going to look at the registry first. So I have an ACR instance. I snagged kubecton.eu. I couldn't believe that that was not taken. To not, I have kubecton.eu.azurecr.io. Super cool. And in that registry, we can see there are three things. The first one is our image. And then the second two are the signatures. The .sig is the Cosign signature. It uses their tag kind of triangulation to figure out how to relate things together. And the second one is the Notary image. And it's using the referrers API. So now we're going to use a tool called oros to kind of look at what that tree looks like. I said it's a directed graph. Let's see what that kind of looks like. So we're going to use the oros discover tool and use the image. We know that it's the Shah 0, or sorry, D09925. So cool. We see the Sbomb and the signature. Those are both of the things that are attached with the OCI spec. So now that we have that thing published, and we kind of know the structure, we know that we're using OCI artifacts to relate those things together, how do we write policies for that? Well, we know we're going to use gatekeeper. We know we're going to use external data. So we're going to need a provider. We're going to write our policies to send images to that provider. That provider is going to do the things that it needs to do with our supply chain security artifacts. But what's that provider going to be? Do we have to write that ourselves? And the answer is no. There is a project called Radify that is specifically built to be an external data provider for gatekeeper that knows how to do things with OCI artifacts and refers and supply chain security artifacts. It's really basic in the sense that it is just a binary. And you use this binary locally. It's pretty cool. You can validate things on your machine. You can test configurations without having to have a Kubernetes cluster, without having to have gatekeeper set up in that whole workflow. You can have an image, and you can validate that image with Radify and configure the policies how you need them, and then deploy that to your cluster. It also runs as an HTTP service so that you can plug into that gatekeeper external data flow. But it works pretty well. And the way that it works is providing a framework and then a set of pluggable verifiers. And the verifiers have a really simple interface. Can I verify this thing? And then verify this thing. And they get, you can see in the signatures, these OCI spec references. So it's going to get a descriptor. And the descriptor is going to have the media type, or the artifact type. And then it's going to have the referrer, or the subject, of the thing that it's referring to. So it can go and do that kind of triangulation and figure out what it needs to do and pull the things down. So you can build a ton of cool functionality into these plugins. Gatekeeper, or sorry, Radify has a few of these already, and we'll take a look at them now. So first, we said that we signed those images with cosine. So let's use this cosine verifier. So we can see in plugins under verifier, we have cosine, right? It's going to apply to vend dev cosine artifacts.sig version 1. And then we make a policy that says we want to apply this to any of those objects. So anything that's in the request that has that type, we're going to apply this to it. You could write some filtering and customization there as well to tailor it down, but we're going to apply it to everything. And let's take a look what that looks like. So first, we're going to use cosine to verify that this is actually a valid image. So we're going to use the cosine tool. We're going to give it that reference and configure it. This is using brand new version of cosines. I have to give it a couple of extra options. But when we do that, we want to assert that this thing was signed by our GitHub workflow. And we use that as the authority that we want to sign against or verify against. And that worked. So that's cool. We know that the image signature was valid. It's successful. Now we would expect Radify using that other configuration to do the same thing for us behind the scenes. So we're going to run the Radify command directly. You can see that. And it returns back that verification report. So we can see is successful true. In our policy, we're probably going to write some rules to say, if the verification result is true, go ahead and send that across. Now we can do the same thing with Gatekeeper. In this case, we'll use notation just to show the difference. But here, again, we have an external data policy. It's going to use that external data provider, which is Radify. We're going to send it the images. And then we're going to apply it to pods in this case. So like the last one we saw, we used namespaces. We wanted to verify the labels. In this case, we're going to use pods. So we want to make sure that all the pods that are coming in in the default namespace have signatures associated with them. And that looks a little something like this. So we're going to take our existing application and then some random internet thing we pulled down from web and networks. We're going to try to run those things. So we'll Coup Ctl run that first. And we can see that the first one was rejected because verification failed. That image was not successfully passed through our policy. The second one is going to be that image that we published to kubekan.eu.azurecr.io that went through our whole release workflow and has all of the signatures and stuff attached to it. So here we go. We're going to run that. Oh, I spelled something wrong, so I have to go back and fix the spelling. Even in recorded demos, I messed up. OK, so it's going through that workflow now. And cool, we see pod created. It did not fail. It didn't get rejected by that workflow. So those are two real simple use cases, right? Those are built-in functionalities that ratify has by default. What if we want to do something more complex? What if we want to wrap a salsa verifier? We produce salsa providence for our build. We want to verify that before we admit things into the cluster. Or maybe we want to use triby to scan an image. Or maybe we want to parse through the serif result from a previous triby run. Maybe we want to invoke some artisanal tools we have internally or we have some government-compliance things that we need to run and validate before we can finish deploying into a cluster. And the way we're going to do that is by building a custom verifier plugin. So those things are built-in, the cosine verifier, the notation verifier. What we're going to do now is take a look at what it looks like to build and deploy a custom verifier. So in this case, what we want to do is block licenses that we know are bad for our business, maybe like AGPL. And then we also know that there are certain packages, log for J as a good example, that we just don't want to allow into the cluster because we know they're bad. So let's go take a look at what that looks like. I'm going to jump out of here. And I'm going to change my display settings just so it looks better for everybody. OK, so in VS Code, what we're going to do is take a look at, if I can find VS Code. Yeah, here we go. I'm trying to find VS Code. There we go. OK, so let's make that bigger. So what the plugins do is implement a really simple interface. They are implementing that, can verify and verify those functionalities. But in the end, they're really just binaries. So they're a binary that ratify will know how to load because you're going to give it this configuration. So we have a name for the verifier that we want to use, the things that we want to verify against. So just like in that config file we saw for the cosine verifier, we said that we wanted to use that cosine signature format. In this case, we're going to use application slash spdxjson because that's what we've published the sbombs as. And then we said that we wanted to use certain licenses as things we want to disallow. And then we want to have certain packages as things we want to disallow in the cluster. So we'll provide those things as parameters here. That's what that spec configuration is going to look like. But the really, really cool thing in here is that we specify what the source for this plugin is. And that plugin itself can also be an OCI artifact, which could have signatures on it, could have sbombs on it. So you can extend this out all the way through that kind of long tail of this whole workflow. So what we're going to do is we're going to build and publish a plugin. We have one written already. It's pretty simple. We'll take a look at it. But we're going to publish it to this ACR instance, which is the same one we've been pulling our images from. And then we're going to register it in the cluster using this CRD definition. So we're going to create a new verifier. Ratify will take it and run with it. What that thing looks like, obviously, we want to define what the parameters look like here, that kind of schema. So we've got a pretty simple set of structs in here, our plugin config. It has a name. It's got disallowed licenses, which is just a slice of strings. And then some packages, which is a slice of a package type that has a version and a name. So pretty basic configuration. And then in the main, the important things to see in here are that we're going to do that verify reference method. That's the contract that we have to satisfy. And at the bottom, we're going to just basically loop through and look for all of the bad packages. And if we find those bad packages, we're going to generate an error message or a status message to send back to Ratify. So basic simple code. Let's take a look at what building that looks like. So I'm going to go back over here. Here's my package or my project, right? So I've got my YAML file that we're going to deploy in just a second. I've got my code. Let's build it. Let's see if this works. I like doing live things. OK, so we're going to build this on a Mac, obviously. So I need to make sure this is a Linux image, because I was going to build it as a Darwin binary by default. And we're going to disable Cgo because this is going to run a distrelas container later. OK, cool. That's done. So now what we're going to do is or us publish that thing. And we don't need that command on the end or that artifact type. So what we're going to do is or us push this thing to that URL, that registry, version 0.0 alpha 0. Oops, I spelled or us push. Run that again. Spell that ratify. OK, so now we're using the or us command to push that, just like if we're going to do a Docker push. This is the same kind of command. It just knows how to work with pretty much any generic artifact that you want to push to a registry. And this is running a little slow because I'm on Wi-Fi. But cool, it worked, right? So we have that in the registry now. We could do an or us discover on that thing just to see if it's there. Cool, and then we could do or us pull on that, just like if we're going to pull a Docker container. Same basic principle. Cool, so it downloads that. Now we have ratify package checker. It's good to go. It'll take a second to download. OK, so now let's jump to one more tab real quick. And let's take a look what's in our cluster. So we've got a kind cluster here. We have gatekeeper. What's running in that namespace? We have gatekeeper already. And we have ratify running. That's great. So let's tail the logs on ratify. So we've got our ratify instance running. That's great. So what we're going to do now is cancel that download. I'm just going to take a little bit. And we're going to do kubectl apply package verifier. Cool, it created a new verifier for us in the cluster. And we can see that it is responding to that already. It resolved the manifest for it. And it's downloading the plug-in just like I was doing with or us poll. It's going to do the exact same thing into the cluster. It might take a little bit of time to download because the download speed is not great on this Wi-Fi connection. So let me just switch kubectl context back to a different cluster. Just a demo magic here. We're going to switch back to our coupon demo cluster. And here we're going to look in that directory. And we'll see, hey, all of our things are there. So that's great. What we're going to do now is check that. Let's get the verifiers. And cool, we see our notary verifier, which is the default one is there. And then our new package checker one is there as well. Perfect. So we should be able to run an image, our image, and see if it validates that or not. So let's do that. Back to this project. We're going to run this command right here, kubectl demo. And we're going to run that image that we built and published as part of our build pipeline. OK, so clear that out. OK, so kubectl run, we're going to name it demo. And then again, the same image we've been talking about through this workflow. And when I run that, you can take a look at the logs and see what's happening in the ratify. Oh, it worked. Ah, it failed to download the plugin because it's trying to redownload it because the ratify is too slow in here. So boo, that failed. We can see that it is trying to use the plugin to pull, using that verifier config, to pull the plugin down, store it inside of the ratify pod that's running, and then use that verifier to admit or deny the workload. In this case, because it failed to download the plugin, it wasn't able to apply the plugin, and it didn't fail for us. But let's see if the other cluster had that. Same problem. OK, let's get our pods again. OK, I think it's still downloading. That's a bummer. OK, so if we try to run this again, I think we'll see the same behavior where it's going to allow it. Oh, no, there we go. Oh, we have a different policy. That's great. That's unexpected. OK, so let's take a step back and look at the config real quick. We can see that's still downloading. It's a bummer. OK, so in VS Code again, the verifier result is going to either return false for a successful, and then the list of error messages that we expect to see come back. So disallowed licenses, disallowed packet versions, if it matches any of those things. Otherwise, it's going to successfully validate things and return those back in. The configuration itself, again, we're going to use that OCI artifact that we pushed to the registry. We can see it pulling it and downloading that inside of the ratified pod itself. And then the configuration that we want to pass into it. So you can write pretty complicated things here. If you needed to combine different things, you could actually compose those plugins together to make different things. But I think the really interesting thing is you can make multiple verifiers. So as policies or needs evolve, and as compliance requirements evolve and change, you'd be able to either consume existing verifiers and add them into your configuration and evolve that as necessary, or write things to satisfy your own business needs as you go through things. OK, I'll jump back to my slides real quick, because I've got one thing of notes that you can take a look at. All right, so what we saw today were a few things from Gatekeeper. So I've included some links to that Gatekeeper documentation. Also a link to the ImageSpec and OCI. So if you're interested in learning more about what's changing in the OCI 1.1 spec, you can take a look at things there. If you're interested in writing your own plugins for Ratify, there's a great link to some documentation here. I used Azure Container Registry for most of this demo, but you could totally use another CNCF project called ZOT. ZOT has implemented the OCI 1.1 spec, and they were super quick about updating it. So as new RCs come out for the OCI 1.1 spec, they're implementing that, and you can totally take advantage of all of these great new capabilities that are in the OCI spec. Finally, I use the ORUS CLI tool. So if you want to take a look at that, you can go to orus.land, and it'll give you all the cool information about that. And then if you want to take a look at my demo, I uploaded it to GitHub, so github.com slash my name slash ratify package checker. And you can download the source and check it out there. Cool. If you want to rate this session, please do so. There is a QR code here that you can scan. You can provide feedback, and we greatly appreciate it. And thanks to our sponsors for recording this and sending it out, and thank you for attending. I think we're out of time, so I'll stay over here if you have questions. You want to come over and ask anything. Feel free to ping into the channel in the CNCF Slack as well if you want to have any kind of discussion there later on. Just feel free to app me at my name, Jeremy Rickard, and then I'll see your message in there. We can continue the conversation.


Internet scanning or how to become your ISP. Please give a big round of applause to Johannes Klick. Can you? OK, that's perfect. Yeah, welcome to the talk. First, internet scanning, challenges, and new approaches are how to become your own ISP. My name is Johannes Klick. I'm concentrating on internet scanning. And my second favorite topic is industrial control system, OT hacking. Have discovered in the past several vulnerabilities, industrial control systems, and the firewalls that have protected them, and also have given some talks to PhDs. That's Russia and in Black Hat USA about how to find industrial control systems on the internet and then to compromise them. And also I did some publications on academic conferences like the ACM Internet Measurement Conference. So what is the motivation for the talk? How to set up an infrastructure for scanning the internet repeatedly? So my problem was that Shodan IO does not provide raw data for free. And as a researcher, you are very interested in raw data because you cannot be sure that maybe comprehensive data or manipulated data you will get. And the next problem was that Shodan IO does not provide clean snapshots of the internet. This means they are scanning the internet, but they are not deleting old results. They are putting every time new results into it. And this means that if you have in your think on dynamic IP addresses, you will have the same server twice or whatever, however multiple times in the database. This means you have no real clean data. Census is very good. Also academic projects are in the past. Now they've got a startup. But also since they are in our startups, they are not providing the raw data for free. And they also have some inconsistency in the database. I will show later. And the next point is you need to think about that people or the persons that are running these platforms, they know what are you looking for. And the question is, what are they doing with this data? And the next question is, who might be interested in this data? So that's why I better to scan on your own internet. Here a good example from Shodan. Here you can see for Ternot that Shodan says you have a property of 5 million Ternot hosts. You're looking at census. They say, oh, it's approximately 3.1 million. And also our own scans show that we have 3.1 million Ternot hosts on the internet. If you there, you can see that Shodan has much more systems in the database, caused by the dynamic IPs and not providing clean snapshot scans on the internet. That's why if you're clicking on Shodan on some hosts, you'll see very often that they are not reachable and so on. You maybe know it when you're working with Shodan. On the next point here, you can see inconsistencies on census. On the left side, you can see I look for, please give me the number of all HTTPS hosts that has responded on port 443. On the right side, I did a similar request. I said, please provide me all hosts that had given you, via HTTPS 443, a status code. Status code is that 200, OK, 400, 403. You know what I mean? And you need to get a status code for a full handshake on HTTP. So both requests should result in the same number, but they are differing. On the left side, you have 41 million. On quite the equal request, you have 35 million. And there's a big difference. This shows there are some inconsistencies, and that's why I cannot use this data for my research. And I have no the raw data access. And that was my final motivation. There was in the first quarter of this year, there was a new project called Packertel. Packertel are some researchers that said, yeah, there's no raw data. We are scanning the internet just using Mascan, scanning the internet from one single source IP and doing it in a time span of two hours. And then we are providing the data for all for free. And then they got some problems with SpamHouse. Because SpamHouse is very crazy. They have a lot of black hole IPs on the internet. And then if they receive one or two sins, they are saying, oh, you're a really, really bad botnet.  And then you're putting it on a spam list and then block list and so on. And that problem run Packertel. Because they are ISP said, oh, no, here, you're virtual private.  Virtual private server, we will kill it. You will not get there. You cannot scan anymore. And there were also some further stories about Packertel. And then they tried to make SpamHouse angry. And in the end, Brian Krebs docked them. And then they stopped their service. Very fun story. OK, so this talk is about how to set up an infrastructure for long-term repeated scans. I'm very interested in the distribution of vulnerable or exotic network services. For instance, like industrial control systems or building management systems in the internet over the time of different autonomous systems or BGP prefixes. And this talk will explain how to set up this framework. In the first 50%, we will talk about the setup itself. And in the next 50%, we will talk about the results of our scans. So what is history? First try was, like every other body has did here, probably in the room, just taking a fast scanner and running it on the internet from a single source appeal. But that's a really bad idea. You will get blocked very first. And we'll receive a lot of abuse messages.  And you are not lucky in the act. Second try, just rent like 20 virtual private servers for four to 10 bucks per month, maybe globally distributed. The results might be getting better because you're using more source appeal addresses. But nevertheless, you have a lot of big problems of abuse messages. Because your service provider will receive abuse messages. They say, oh, you're malicious. You're infected.  Please clean your virtual private server. Or you will get kicked out like the Packetel guys. The third try was to rent a slash 29 from an ISP to get your own who is DB entry with your own abuse mail contact. Then we thought, OK, we are lucky. Now everything will be fine. But the problem was that these automated IDS intrusion systems, they are not only reporting to the abuse mail that's standing in the US entry. They are also sending messages to the maintainer of the IP address block. This means you're direct ISP. And then we got the same problem that our ISP got informed that we are malicious. We have a botnet and whatever. And then we got a lot of trouble. Yeah, well, in the end, we managed a lot of abuse messages on our own. But in the end, it was too complicated to get some discussions with the ISP. So the final solution is we got our own ISP. So we become a RIPE member and get our own autonomous system, a complete slash 22 network. This means 1,024 IP addresses, sign up fee is approximately 2,000 euros, membership fee is like about 1,400 bucks. And then we ran two different co-location spaces, got a further slash 29 from another ISP. And then we bought a server for 30,000 euros for all the big data management. And yes, and then we used auto replies for our abuse emails. This means if somebody feels annoyed by our scans, he sends a abuse message and get an automatic reply, which explains what we are doing, why we are doing the research, and how he can and what is the way to get excluded from our scans by putting them on our blacklist. So the result is that the abuse messages reduced by approximately 90%. And that also the IDS messages that told us that we are get blocked and so on, also massively reduced. And we can now do complete abuse messages of our own. And this shows that using a slash 24 server, like 1,000 IP addresses for source addresses for scanning works really good. So what is the infrastructure? So at first, we have a front end. Then you can click OK, I want to scan this IP address space. Very often we are just saying 0,0,0 less, 0.  Then this goes to the scan master. The scan master looks what scan algorithm we have chosen. Then he will take the IP for address space, cut it into small pieces, randomize the IP addresses in the small pieces, and then it sends to the search node clusters, the search node cluster scans the internet, and sends the data to the aggregator. Furthermore, it is very important that if a search node fails, that the work package, the search node has to work on the number of IP addresses, we'll get reassigned by a scan master to another search node. This is a kind of fail safe, because very often maybe people using ZMAP or distributed ZMAP scanners, they know it. Then a ZMAP server node got broken, and then you have to do some manual work, and then you have to choose to look on what was the last IP number, and how was the last prime number, and how to, in the end, go on with your own scan. And that's very important. If you want to set up a global scan infrastructure, you need a kind of scan master that is managing the scans and looking that everything is working. And if not, then you need to have a kind of recovery procedure. So the search node cluster itself consists of different search nodes and two different co-location centers. At first, we have a SIN scanner, just sending the masses of SINs out on the internet, itself written to go. And then we have an app scanner. It's a customized ZGREP version, because ZGREP is a very good tool that supports a lot of protocols. And we added some further protocols and used these kind of application scanner. And then all the data goes to the aggregator, the task of the aggregator is that if a search node failed and another search node has taken over in the ANSI work, then you have maybe multiple results. And the aggregator aggregates all data and pays attention that you will not have multiple results in the database. Then you have data enrichment. That's very important. Just scanning the internet is not sufficient. So maybe you have all the ports, and then you have the application data, like TLS, certificates, information, and so on. But you have to enrichment with further data. We have access to the Whois database. So the internet objects, this means if you're doing Whois 8.8.8, you will receive all of that. It's a company that owns the Whois blog, and then maybe send some information for what is the Whois blog used. Then we have also reverse DNS, Jiro IP, let's say, where is the IP approximately located. And we're using a BGP prefix information. So we're saying, OK, to which announced BGP prefix, does this IP belong? And also, we put the information together which autonomous system belongs to the IP address. As storage solution, we have chosen Elasticsearch because it's one of the best databases I have ever seen. Recently, since the new version 7.3, it's very good. And here, you can see that as a protocol. So we are currently support. We are concentrating very strongly on industrial protocols. But we also support the normal protocols. Most of the protocols are from ZGREP itself. But we have also some special protocols like the Phoenix protocol and some other. So now, thinking about some numbers and settings. So if you really know it, IP addresses, we have approximately 4.3 billion IP addresses. And very often, people are sending all the IP addresses out, but the problem is that only 2.8 billion of these IP addresses are rooted in the internet. This means you can only reach at a maximum of 2.8 billion IP addresses on the internet. So if you get some kind of BGP feed, like if you have your own BGP router or have a feed 2G BGP router or use QIDAR data, you can reduce the number of TCP SYN packets by 35%. Because when you only send the TCP IP packets in the rooted IP space, then you can save a lot of TCP SYN packets. You don't need to scan the private IP addresses and so on. So first thing you have to reconsider is that a TCP SYN packet is approximately 70 bytes big. This means 70 bytes multiplied by 2.8 billion IP addresses means that you're just sending data with a single TCP IP scan, 200 gigabyte of data you will send into the internet. But the interesting point is that you will at the maximum get only reply from 2% of all IP packets you will send out. It's a very easy calculation. The most used protocol on the internet is HTTP. And so you can see that we have 56 million HTTP hosts divided by 2.8 billion rooted IP addresses. It's 2%. This means you have just a hit rate of 2%. 98% of all TCP SYN packets from the 2.8 billion IP addresses are overhead in the app. So next question is using one SYN or two SYN. Every time it's a discussion, we have tested it. Sending doubled SYNs makes no sense because the results get not doubled, right? You see that's neglectable in my perspective. You maybe receive some better results, but we can also see on SSH you are able to receive worse results. For me, it's a standard deviation. And also compared to sensors, you can see that the results, our results, are really comparable to sensors. That these kind of setup works. Very interesting. Senses uses slash 23. We are losing a slash 24. This means we have S2, SOS IP addresses, and our results are also some minor better. But you have to keep in mind sensors is running much more longer as a service. They are scanning much more internet, and they have maybe a bigger blacklist. So what about Skeet? Very often people say, may I scan maximum speed? I can't don't recommend it. If you use the packet data they provided it, you can look it up. Then you can see that we are scanning with ScanSpeed at approximately 70 Ambit, taking 6 to 7 hours. And then you can see that if you scan from a single SOS IP like everybody can do in a time period of 2 hours, your data will be degraded by 10% to 30%. This means you will see, in the worst case, only 2 third of the whole internet. So scanning strategies, very interesting point. You see that imagine a new zero-day remote exploit over a zero-day arrives on Earth, right? And then you're maybe interested to scan the internet very repeatedly, not like every week, but like every day or every 10 hours or 20 hours, whatever. Then it's a very good idea to scan the complete internet once and then to re-skin only the BGP prefixes with at least one host in it. And this will save you additional 25% to 50% of the routed IP44 address space and scan time. And I call those BGP prefix hitlists. I've written a paper about it published on the internet measurement conference. If you're interested, please have a look into very interesting. So next problem was we are our own ISP. And suddenly the police arrived and said, ooh, you're doing malicious attempts. No, they do not ask us for doing malicious attempts. They said, who is the owner of this IP address? Who has used this IP address? And then I, because we are our own internet service provider, and then we receive it. And then we talked with them and said, now we are the owner of the IP address, and we have used it for the internet scans. And they got some kind of afraid because normally you are not allowed to get informed that there are some police investigations about you. But then we explained that we are a T security researcher, that we are doing good stuff, that we are providing the information, that we're informing people, and so on. And then for them it was OK. And we have never heard about it. But in the first moment, I have read this email. I got, well, it was really amazing. Hardware setup, very important if you want to do a lot of internet scans, also with protocol application, data enrichment on zone. The size of the protocol is quite different. And you can see it's already really big numbers. A single HTTPS scan is about like 700 gigabytes. HTTP is 300 gigabytes. And Ternot is only like 2 gigabytes of data. So therefore, we have bought a big server consisting of an MD-APEX 7551. So we bought two of these CPUs. We have one terabyte run. And we use 50% of the memory for the elastic search heap itself. And we use 50% just for caching. Then we have 40 terabyte of SSD data because the stuff that cannot sit in the RAM needs to be stored on the SSD because we need a lot of high IO interaction. And then we are running it on a rate null for just performance increasements. And for long-touch storage and also copying the data from the index that they're running on the rate null SSD, we have about 72 terabyte HDDD running on the right file. So our first problem was when we have gated our server, we started Htop. And we were not able to see our processes because we got so much CPUs. And they got hyperthreaded. It's like 121, 128 CPUs. And first, we need to investigate how to configure our Htop to see our processes. OK, next interesting point, elastic search. Also, some tweaking points. Most people do a really, really big mistake. They have maybe a lot of memory, and they're doing just only one elastic search node. But the problem is, if you have your node has allocates more than 26 gigabyte of RAM, then the Java virtual machine, they use 64-bit pointers instead of special compressed 32-bit pointers. Java does some pointer special magic. And the problem is, when they use 64-bit pointers, then each load of a pointer uses as much as twice of the bandwidth, because 62-bit are much bigger than 32-bit pointer, right? So this gave us, so we have divided elastic search into different search nodes. And each node has a maximum of 26 gigabyte per node. And this has given us an increase of approximately 30% to 30% unbelievable by just splitting the database into several processes in the end. If you're interested how Java is doing memory management and so on using elastic search, there's a very good article about it. Just have a look into it. So now let's go into the results. So what we see here, it's a Kibana. It's the front end of elastic search. And here we have interactive possibility to get a rough overview about the autonomous systems and the structure of the internet. At first, you can see that the most HTTP hosts here are hosted by Amazon. And then we can go see what is the next biggest autonomous system? OK, we see Akamai. Akamai is one of the biggest content delivery networks we have on the Earth. And then in the second ring, we can see the BGP prefixes. There you can see that Amazon has here a slash 12 and announces this area, also a slash 12. There we have a slash 13 and so on. And the next step on the third ring, we can see, OK, that is Amazon. We can see that it's this specific, announced BGP prefix, 3208 slash 12. And we can see how is the service distribution inside of this prefix. And that is the way how I understand the internet. The internet is not about nations and countries. It's only about autonomous systems, their prefixes, and the internal structure of the prefixes. Which kind of services do they use? And here, you can very good see that they use Apache, NJigs, and also some other servers. And you can see there's a big difference between Amazon and Akamai. Because here in Akamai, you can see every server in the prefix has the same server banner. Every time it's Akamai G host, you will get any information about which kind of web servers are they using. Is it their own web server, is it Apache or NJigs, whatever? They have just changed the banner. And here, you can see the difference between two autonomous systems. Akamai is content delivery provider. Amazon is more kind of a hoster, right? And that's why they have a much more heterogeneous structure. What I have now done is we clicked on Amazon and said, okay, let's look into Amazon. How is Amazon structured and doodle? And here, you can see the different prefixes. And again, the service distribution, you can get the rough overview. It's just about, it's a top 10 or top 12 prefixes. And now, we are adding the who is information. And now, it's getting very interesting. So we have, from the inner circles, the autonomous system. Then, we have the prefix. Now, we have the who is description and see who is prefix. And we can see that Amazon is structuring their network. They have a slash 12, and then they have inside a slash 26 standing in the who is database. And we can also see the customer. Here, it's actor incorporation, right? And then, you can go around and can see about the different Amazon customers and see how is Amazon structuring their internet. And then, you can look, okay, that's the BGP prefix, that's customer XYZ, that is prefix, who is prefix XYZ. And then, you can look how is the service distribution inside of the who is prefix. You are cutting the network in more slices and getting a good understanding about the structure of the complete autonomous system. Here, we can have found Zoom. Zoom is a very well-known video conference communication system. And here, we can see which kind of service they use. We will click on it to just say, okay, I'm interesting, how is Zoom that is hosted on Amazon? How is it structured? They are using a lot of services just called Zoom, but they also use some 1.37 service from NJX and also some NJX without any kind of version information in the end. So, and using autonomous system information, BGP prefixes and also who is information gives you a very good overview about the structure of autonomous system and about the internet itself. So, going next, here, another service that was backup. Now, here, a look into OVH. OVH is one of the third or one of the biggest European hosts that we have. And here, we can also see the BGP prefix. Oh, let me go to the presentation mode. So, here, we can see they use this slash 16 who is says it's for dedicated service and then we can see they use the slash 18 from the slash 16 for dedicated service. Somewhere else, we can see they use the slash 24 from another slash 16 for failover IPs. This means we get infrastructure information. How are organizing themself? That's maybe interesting for an attacker or for somebody else, right? So, and then we can also see which kind of services do they use. And we can see that all failover IPs have the same web server in the end. And what we can also do is which kind of web services does OVH at all use. And we can see mostly they use Apache or Android, Apache and so on. So, we can see here is a separate list sorted from top down by the major numbers. Very interesting. But then we have a complete another autonomous system. Here we have SSH, a complete another protocol and we can see that GoDaddy does not reveal any kind of information about the internal structure. All who is blocks are the same and they also have the same name like the autonomous system itself. And very interesting, they are using everywhere like on 90% of all their servers, they are using the same open SSH version. So, you can see that GoDaddy is a hoster, right? So, there is a web hoster. So, they are completely managing different on a different way than OVH or Amazon is doing, or HACMA. So, using this, we have looked for different customers. And if you look in the history description here, you can see you can integrate a lot of customers from Amazon. It's very interesting. You can see we have Cisco, you have Samsung, you have Zoom and so on. Next, we can also look how is Amazon structuring their EC2 infrastructure. That's all the cloud stuff and so on. We can see we have a lot of hosts in AVH Asia Pacific because they have a very big slash 16 there. And we can see that they use airport abbreviations for the locations of the EC2 clusters. So, we can see it's Dublin. Then we can see they're using CDG, which stands for child to go, it's France. Then we see FRA, it's Frankfurt, it's Germany, and so on. And that's very interesting. Using also the US information, you're getting a lot of infrastructure information about the autonomous systems and of companies, how are their structures, the network, and for what parts, and you know what they are doing with their subnets. Then, you can also do some other further interesting analyses. I have said, please provide me all Microsoft IAS 5.0 servers. Who knows what is the operating system of it? It's Windows 2000. And then I told my database, please provide me all the subject common names. So, we have a list of the subject common names and we have now a lot of domains that are running on a Microsoft IAS or Windows 2000 server, a very, very old system, like 20 years old.  But then we can see here on the right side, it's the start date, the start validity date of the certificate. And there we can see that our servers, they are running for some stuff because all dated forgotten servers get not new SSL certificates, right? People are running Windows 2000 servers, 20 old software, not supported since 15 or 10 years, whatever, and providing them with new certificates. And that is unbelievable for me to see that people are doing it. And it's also awesome that these Windows systems are still running, right? Okay, now having looked at the dark side. What we also can do is that we can, we also doing some help lead analyzers. And I told my database, please provide me an overview about all governance agencies, whatever, in Austria, and the HUS database, and that contains a TLS certificate from Fortinet in the end. So we are looking for Fortinet Firewalls that are hardly vulnerable in Austria on governance agencies in the end. And we can see here that we have found a lot of, so it's German and it stands some like a bureau of District One, District Two, and whatever. And then we can see that the issuer organization keyword is Fortinet, so the issuer of the certificate is Fortinet. And on the right side, you can see very good the different versions of the firewalls. That was scanned eight from 2017. Then I scanned again at the internet in one year later, and now you can see they have replaced the Fortinet Firewalls with Sophos appliance, right? Very interesting what you can see using time differential analysis in the end. So then if you're scanning for industrial control systems, you can also see that that's a scan for a seven common modbus. You can see a very big distribution, and you can see there's a hard correlation for the economic point. So if you look at the former GDR region here, you have not that much industrial control systems. You have more in the South and the West. You have in North Italy a lot of industrial control systems, hard, big economic area in Italy. In South Italy, you have not that much industrial control systems. Same to Scotland compared to England. And it's very interesting that you can derive the economic power from the internet scans. Very interesting. And we can see it's a very big problem that over the world, people putting industrial control systems on the internet. Here are some numbers. So it makes sense as seven common stands for Siemens, and we can see Germany has the most number of Siemens industrial control systems also fully reasonable. So now next analysis. Here we can see, I said, I want to see for heartbeat vulnerable servers in April 2017 in Afghanistan. And here we can see on the autonomous system, the autonomous system is the Afghan telecom government communication network. And see who is description is, and that I have never seen and so detailed who is information. It's the government communication network, district communication, whatever, for the communication of 34 provinces into 357 districts in Kabul, Afghanistan. And now you're looking, who is the issuer of the certificate? You can see it's a PF-sense firewall. So you can, if you're doing these kind of analysis and putting data together with other data, you're very fast finding very interesting results. And I'm sure that the CIA is going on since some years, right? Okay, now we are getting to the real bad point. People are putting in the who is information, a lot of stuff. They also say, this is power plant XYZ. I can't believe it, people are doing that. And here we can see it's a bigger power plant, it's a coal-fired power plant. And here we have VPN endpoint via HTTPS. And this power plant has a capacity of approximately 500 megawatts, supplying like 1.5 million people. It's not a small power plant, right? So, and now we are looking, the scan date is from 2018, right? So, August 2018, and we can see the certificate got invalid on July. Okay, it's one month, it's okay, but in the end, it's a critical infrastructure with an invalid VPN endpoint certificate. Then I did the rescan, I reformed them, I emailed them, I contacted them, I got never a reply. And then I rescaned November, and still again, the certificate is out of debt. And now I did some new scans and I found that they have updated certificate in April 2019.  So, for me, I'm living in Germany, I'm not funny with that. So, now I will provide you another further, very interesting analysis. Another power plant, where is it? Perfectly. So, what we have here in the first is we are looking for Kraftwerk, that stands for power plant in German. And we can see, we have only here, the autonomous system name, the BGP prefix name, and the server banner. That is the information you can see on sensors or showdown because they do not use who is information. Now, we are, say, please enable who is information. And then we can see some very more interesting result in the end. So, let's enable the who is information, and then we can see, we have RVE, it's a very big German energy company, and the power plant called Niederhausen. Looking at Wikipedia, what is Niederhausen? Oh, it's just the second largest coal power plant in Germany. This time we have about like four megawatt of energy capacity in the end. So, now that's very interesting, but what is this for a service? Oh, very interesting. It's a Lancome 7111 VPN gateway. Lancome, hmm, for critical infrastructure, okay. Let's look. Ah, there is a list of services devices, right? Maybe it's the end of service device, maybe it's like one year old, I don't know. Let's have a look how old is this unsupported system, or since when is it not unsupported? And here you can see it's Lancome 711 device is unsupported since May 2010. It's a nine year old unsupported VPN gateway, and that is again from the mid of August. It's like one week old, right? So it's a reason we can see it on the internet life. So, very interesting. Thanks. So, now it's getting better, right? Let's look which kind of cipher suite is it supporting. So now let's go back again to the Lancome device and we can see oh, it's the secure cipher we have on the world, it's the TLSRC4 cipher, right? Oh, unbelievable, because it's so old system, it's not surprising, but in the end how? We have a critical law here in Germany, energy companies spending a lot of money. How could it possible there's a very, very 10 year old VPN gateway with a RC4 cipher on a critical infrastructure? I cannot get it, right? So, I have a lot of data. I could talk like 20, 30, further minutes, whatever, but it's my wife's sense I have to stop spending so much time on this data. Yeah, so now we are going to the summary and then I'm very, I want to discuss with you about the results, okay. So, summing it up, using raw data of internet scans is very important and you have to enrich with BGP who is in protocol specific information.  There are a lot of, it's very important for you to understand the topology, the structure itself, and you can be very easy for you than in the end. If you do your own scans in data enrichment to identify all websites or vulnerabilities that maybe belong to a company, to a critical infrastructure or maybe to government agencies in the end. Furthermore, how to do this? Here is my small cheat sheet. Get your own AS versus LESH-22, this means like 1,024 IP addresses. Scan not faster than 70 Ambit, use once in, two in, makes no sense because you take like twice for your scan and you will send too much data and you will get more engaged with IDS systems. Scanning the internet in two hours from a single source IP, you will only see 10, you will only see like 70 to 90% of the internet so it's not a good idea. Or you need, you can do it but then you know that the data is not very reliable. You can use only, you should only scan BGP prefixes that are routed and that way you will save about 35 of your ZIN traffic and time and if you want to do very fast, repeatedly scans in case of a short remote executable vulnerability that is maybe released, you should use BGP prefix hit list meaning scanning the internet once and then only scanning the BGP prefixes and this will save you further 25 to 50% of the routed IP for for address space. Thank you very much. If you want to email me or whatever just email me and I'm living here on the camp, it's a garaffle and a big re-intent and if you're interested in scanning, please contact me because a lot of people, I know what scanning ones are self and they're not talking to each other and that's why I'm sitting here. I want to explain to you how to scan and I really would like to set up a kind of scan community in the USA they have a scanning community and here in Germany we won't. So please help me come around with a beer and then we can talk about how to scan and I have also some further interesting informations. Yeah, thank you very much. So thank you Johannes for a super interesting talk. We have some time for questions. So if you have any questions, please, we will have a microphone shortly but in the meantime we'll take a question from the internet. Yes, the internet wants to know whether it's possible to access the scan data or where you publish your scan results or your research results. Yeah, so in the end I'm not publishing the results results because this talk was aimed to explain you how you can set up your own infrastructure. So you don't need that much money like 30,000 euros for the server. You can adjust it because I want to scan a huge load of different protocols in the end. So this talk, the aim was to show you how to do it on your own but I want to share the information with penetration testers because they need this kind of information to detect the complete attack service or companies they are dealing with in the end. And it's for me it's to, in the end, I do not know what people would do with all these data and I don't want to be as the people as the guy who was publishing it to maybe to the bad guys and that's why I only want to provide it to some penetration testers. Okay, a question from the audience. Hello, Naif from Italy. So I'm doing a project focused on Italy and doing massive scanning and I come up with an interesting problem that you may already have figured out. Basically, those kind of scan are not aggressive vulnerability analysis are basically creating a software inventory from the service you can connect to and speak to. I came up to the problem of out of a scan I would like to detect the surface of vulnerabilities. So get out some kind of indicators that this service is vulnerable or not. And the very simplistic approach is to make a matching with a CV database or application version fingerprinting but I've been told by multiple person that the amount of a false positive is pretty high. Do you have any insight on that? Yeah, so you just need to look on especially Debian. Debian distribution is doing very often. If there is some kind for vulnerability version 111 of a certain web server, they will provide a page but they will not increase the minor number of the service. So it's only a potential vulnerability in the end. That's a big problem. And just the only way would be try to exploit it but you cannot do it as a researcher and there's bad guys are in a better position in the end. So we can only say what is a potential vulnerability or capacity in the end. Yeah, I'm totally agree with you. That's a big problem. And for this I have no solution in the end right now. Next question. Thanks for the talk. Have you thought about scanning IPv6 prefixes? Yeah, we have. We have some current also research project in it. Maybe next year. Thank you. Any more questions from the audience or from the internet? OK, looks like we're done. So again, a big round of applause to Janis. Thank you.


نحن لا نساعد أي شيء other than Allah سبحانه وتعالى نحن نعتقد في واحد الله نحن نحن نعتقد في واحد الله و الله أردت للهيومين الكثير من المملكين أنهم هيومين و نحن نساعد في الله مثلنا و نحن نساعد في الله لكنهم أردت المملكين من الله they were beautiful people they were very good people جيسيس موسيس إبراهيم and the final one is محمد that you can say the shahada tonight تستموني أن لا يوجد إبراحيم إلا الله و محمد is the final messenger once you say that you are all muslim clear muslim and الله سبحانه وتعالى forgive everything you did before and this is I'm not saying it from my all we have evidence for that from the say in the Qur'an and the say of the prophet like when someone became a muslim and he is clear he's like white paper he's like baby like just burned at that time so that's why we'll ask you to pray for us because you you're clear at that moment so insha'Allah I'm gonna say it in arabic just repeat after me word by word I will go slow yeah please get that good so insha'Allah by the way the word you're gonna say right now we call it shahada the prophet Muhammad said this is the best word the prophet Muhammad said and the prophets before him خير ما قلته أنا و النبي يونة من بعدي لا إله إلا الله محمد رسول الله the best thing I said and the prophets before me they said is شهادة that we're gonna say now insha'Allah أشهده أشهده الا إله إلا الله وأشهده أنا محمد رسول الله ما شاء الله so what did you just say so you said it with your tongue and you believed it with your heart right الحمد لله let's say it one more time all of us together insha'Allah what brother أشهده ألا إله إلا الله وأشهده أنا محمد رسول الله الله أكبر الله أكبر الله is great this is your gift from MCC some books in here and to read insha'Allah and learn more about Islam Welcome to Islam you're Muslim you're our brother like many other Muslims الحمد لله today is the second one here in MCC took شهادة we had early brother and this is second brother give me a hug if you don't mind we love you we welcome you to Islam you're our brother الحمد لله الحمد لله


Hey everyone, Devin Smale here at Mountain View Dodge. We've got a 2015 Dodge Grand Caravan SXT 3.6L V6. This one's got your Stow and Go seating, as well as seven passenger built-in roof rails, traction control, and a decent training. Devin Smale, Mountain View Dodge.