Okay, so thank you everyone for joining us. Welcome to today's CNCF webinar, securing your serverless journey. I'm Sanjeev Rampal. I'm a principal engineer at Cisco and I'm also a CNCF ambassador. I'll be moderating today's webinar. We would like to welcome our presenter Ron Harnick, who is a senior product marketing manager at Prisma Cloud by Palo Alto Networks. A couple of housekeeping items. During the webinar, you will not be able to talk as an attendee. There is a Q&A box at the bottom of your screen. Please feel free to drop your questions again. Note, please do not put your questions in the chat window. Please put them in the Q&A window. This is an official webinar of the CNCF and is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of that Code of Conduct. Basically, please be respectful of yourself, your fellow participants and presenters. With that, I'm going to hand it over to Ron to kick off today's presentation. Thank you so much. It's good to be here. Thanks for the opportunity. Yeah, so just a little bit of context of what we're going to be talking about today. We're going to be talking about an introduction to serverless security. So we'll discuss a little bit about what serverless is at a fairly high level just to make sure that we're all on the same page. And then we'll get into what are some of the risks or some of the things that you have to think about differently when you're trying to secure serverless workloads. And then at the end, we'll do a fun little demo just to understand how one might carry out a serverless attack. So just real quick about myself. So my name is Ron. I'm part of the Prisma Cloud product marketing team. Prisma Cloud is the cloud native security platform from Palo Alto Networks. So over the past 10 years or so, I've been in the IT and networking and security and cloud space moving around between all these different companies. I started out as network engineer and then slowly moved through different roles within the engineering organizations and product before I eventually sold out and moved to a marketing role. So I was with PureSec, the serverless security platform that was acquired by Palo Alto Networks, I think just over six months ago. And then I got the opportunity to join the Prisma Cloud product marketing team. So even though now I get to talk about cloud native security as a whole, serverless is still something that I'm very passionate about. So I'll take every opportunity that I can to yell at my laptop about serverless. So thank you for giving me another one of those opportunities. I live in Israel with my wife and my two beautiful twin girls, which I'm obsessed with throwing a picture of them in everything I do. So here's a picture of them now. All right. So let's quickly go over the agenda for today. And just like we said at the top, please feel free to send your questions through the Q&A panel. I'll try to get to most of these as I go along. But if there'll be any unanswered questions by the time we wrap up, we'll try to get to those in the end. So we'll start off by covering what serverless security is in a nutshell at a high level and make sure that we all understand just generally how a function based application works. We'll focus on a few risks and pitfalls. Obviously there's a lot that we can talk about and we can even broaden the discussion to cover application security as a whole. But we'll very much focus on function based applications and we'll take a look at a few of the things that you need to be on the lookout there. And then we'll wrap it up by doing a quick demo of how we might carry out an attack. You know, I'll tell a fun little story that happened to us while we were at PureSec that I think really helps illustrate the importance of permissions on functions. So yeah, should be a good time. Okay, so the first part of almost any conversation that you have about cloud security, not just serverless security, almost always starts with this idea of the shared model of responsibility. And I'm sure most of you on this webinar have heard this a million times. When you move to the cloud, when you start using something like Google Cloud Platform or AWS or any other public cloud platform, the shared model of responsibility at its simplest form divides into two. The cloud provider is responsible for the security of the cloud. So that is anything like the data centers themselves or anything underneath the virtualization layer that you care about. And you as a user, you care about the security in the cloud. So everything that you put into these workloads, so your code, the user permissions into into these workloads, things like that. With serverless, there's an interesting shift that that happens within that model. So when you're talking about infrastructure as a service, you're essentially taking a lot of responsibility to on you're taking basically a big part of that pie as responsibility for yourself. You own pretty much anything that sits above that lowest level of virtualization. So anything beyond, for example, like an EC2 instance is completely your responsibility. So fixing patches in your operating systems and scanning your images for vulnerabilities, all that kind of stuff, preventing data leaks is completely your responsibility while the cloud provider really only cares about the physical infrastructure and the configuration of like the underlying infrastructure beneath the virtualization. When you go serverless or when you start, this is offset a little bit, sorry about that. So but when you adopt more of a serverless framework, there's something really interesting that happens to the split where the cloud provider actually takes on a lot of responsibility and you take on less responsibility when it comes to what is left to secure. So it almost ends up being like this 50-50 split. And it's interesting for someone in my position because working for a security company, typically you would go up on stage and you would talk about this scary new technology that you have to do all these things in order to leverage successfully. While with serverless, the conversation is more like you just have to think about a few things differently, but your starting point in terms of security is actually pretty strong because now you don't have to think about patching your operating system. You don't have to think about networking or a big list of other things that you used to have to worry about. Obviously there are new attack vectors and new things that you have to worry about and most of the things that are relevant to application security as you know them today are still going to be relevant and that's exactly what we're going to be covering today. So just to make sure that we're all in the same page, I try to make this as accessible as possible to folks who are veterans with serverless and folks who are new to serverless workloads and even the term serverless can be sort of like a point of contention in terms of what it actually means, but for the sake of our discussion today, I'm talking specifically about applications that are based around functions. So around function as a service, services like Lambda or Google Cloud functions, for example. So typically how these applications would work is something like this. You would have a developer commit code to a code repository and then that code will go into a function. For example, an AWS Lambda function. Then that code would sit there and it would wait to be triggered by some sort of an event. So some sort of an event will happen. It will trigger this piece of code. Then the function will do whatever it is it was designed to do. Maybe it will interact with another cloud resource or maybe it will just have some sort of output. For example, in a little bit we'll be covering a serverless application that is essentially used for HR for hiring. So someone sends an email with a PDF resume attached to it. That gets intercepted by a cloud service that then triggers a Lambda function and then that Lambda function takes that file, converts it to text and puts it in database table. So roughly follows the sequence of events that we're seeing here. The core of serverless or the core of these types of function as a service tools that you can use is that typically this will be a very small piece of code that represents a microservice. It will usually not be some massive monolith of code. It's very much towards the cloud native approach of adopting microservices because these functions will typically be short lived. So they'll typically live for only about 15 minutes. So it's really not a discussion of should you go 100% serverless or should you evolve from, for example, containers into serverless. It's more of like a spectrum. A spectrum of compute options where you might have your virtual machines on one side and you'll have something like Fargate or Google Cloud Run and then you might have something like Lambda or Cloud Functions. And every different tool is going to be good for different types of jobs. All right. So if we agree that this is a good baseline for how an application looks like when it's based on functions, let's talk a little bit about what are some of the attack vectors or attack surfaces that we need to talk about. And most of these are going to be familiar to you if application security is something that you've been interested in. Okay. So let's say that this guy is our attacker. Let's focus on what ways, not what what the actual risks are, but what are the paths that he could possibly take in order to compromise our application. So the first and maybe the most unique to serverless that that we need to take a look at is event data injection. And I'm sure we're familiar with injection attacks like SQL injection, for example. But with serverless, you have to remember that there are thousands of event sources that you could potentially use in order to trigger your function. So it's not necessarily going to be an HTTP header that we can be very familiar with with the fields and the cookies and the different parameters within there, but it can be an S3 bucket or it can be a Kinesis stream or, you know, any of those types of things. Each one of those event sources is going to have its own structure in its own format of how it conveys information. And those are things that could potentially be manipulated in order to compromise the function. Another would be unauthorized deployment. So this would be, for example, a phishing attack where someone was able to deploy code out of band directly into your Lambda function, for example, because they got the credentials to your AWS account or anything in that way. Something that's going to be less far fetched and much more common, not just in serverless, but in application security in general, it's going to be dependency poisoning. And dependency poisoning, really what we mean here is, you've used some sort of open source third party library within your application within your code. And there's a vulnerability, a CV or something else within that NPM package or whatever that dependency might be. And this is exactly, or at least part of what the whole discussion around shift left revolves around, like you have to be able to detect vulnerabilities as early as possible in the lifecycle to make sure that they don't make it all the way into your production environment serverless or otherwise. So that would be something to look out for this vector. I just want to add dependency poisoning. It's something that is that exists as an attack vector in pretty much all forms of application security, specifically with serverless, the challenge is going to be how do you detect that? And what can the attack were actually do if they compromised your function? And finally, the last vector that I want to talk about is tampering with data or really compromising the cloud services that your function interacts with. So a typical function based application or serverless application is not going to be made just out of functions. You're going to have maybe a storage bucket talking to a function that's talking to a database table and maybe that is talking to another function, or maybe you have a website where you have multiple different functions doing the login or one is loading the pictures and one is storing all the user data, whatever it might be, there's going to be interaction to a lot of different sources within that process. If one of those other services gets compromised, then that entire application could get compromised. And that's actually exactly the example that I'm going to walk through in just a little bit and a little story that we put together for you. So I think the most famous example of this is S3 buckets that are open to the internet. So maybe your function itself is locked down tight, but it stores lots of data on an S3 bucket that is open to the internet and then an attacker might be able to compromise your function through there. What can an attacker actually hope to achieve with these types of attacks? Well, these are typically the same things that you will find with typical application security. So they might be able to compromise data. So for example, let's say that you have a function that's talking to a DynamoDB table. If that function has permissions to essentially do whatever it wants with that DynamoDB table, once that function is compromised, then the attacker will be able to exfiltrate data from the DynamoDB table that they're talking to, which again is relevant to the example that we'll see in a little bit. So this is just a quick overview of some of the vectors that you might need to look out for with Serverless. So what can you do to educate yourself about this? So one thing that I would definitely, definitely recommend is to read this white paper from the Cloud Security Alliance. This is something that a lot of different experts from different companies came together and put this together, definitely not product related. So it's not a product pitch for any specific product. It focuses on functions. And it's essentially the OWASP top 10 version for Serverless. So this white paper, which is very long and very exhaustive, I would still, if you're considering Serverless or using Serverless, I would very much recommend that you go through it and read it. Just because it goes into much more detail, then we'll have time for today for what are the attack vectors that that you should be worried about or should be on the lookout for? What can you do to hard view your applications? And it also just has a lot of really great insights about how function as a service tools work. So I cannot recommend this enough. And this was done by the Cloud Security Alliance, which is a fantastic organization. All right. So if we kind of likened the security on serverless application to applications to applications security in general, let's talk a little bit about what why we need serverless security or security that is specific or purpose built for serverless. Well, with traditional security, you're essentially protecting applications with tools that are being deployed on servers or networks. With serverless, you don't have access to any of that underlying infrastructure. So for example, if you have that S3 bucket that's talking to your Lambda function, there's nowhere for you to put a firewall between those two. If you're only using something, for example, that looks at an API request at the HTTP headers, that's only going to cover a small percentage of the things that could trigger your functions. So that is typically where you have to think about this a little bit differently. So with serverless, the application owner doesn't have any control over the infrastructure. So traditional security tools, I don't know if traditional is the right word here, but traditional security tools are unsuitable to secure 100% of these types of environments. Having said that, I'm very much laser focused on serverless for this conversation. It is important to remember that in most environments, it's going to be some sort of a hybrid, right?  So security is a much broader conversation, but specifically about how do you secure, for example, a Lambda talking to a DynamoDB table? Yeah, you're not going to be able to do that with an agent deployed on a server or or with something that inspects anomalies on the network. Let's take a look at how typically you would protect an application. You might have your application code and you might have something for endpoint protection that takes a look at the behavior of the application itself. Maybe you'll have some sort of IPS type tool that takes a look at network anomalies. You might have a secure web gateway that takes a look at outbound traffic or you might have some sort of firewall that takes a look at inbound traffic and you'll have maybe even an external firewall taking a look at your your layer seven traffic and all of this goes on your infrastructure. So this is a, you know, a 360 view of how you lock down that application very, very tight with serverless. Pretty much all of this goes away and your functions essentially live on the internet. And that's something that you have to be on the lookout for, right? So and this is where to me the conversation about serverless security gets really interesting because the code that you're deploying, it doesn't live within a container. Well, technically it does, but I mean, it doesn't live within some sort of structure that you deploy to the cloud, but it lives directly on AWS or Google or Azure or wherever is you're using your fast server. So with that, the entire concept of security is pretty much reduced to strong code and strict configurations. So what can each function do? What permissions does it have? Obviously, there are other tools out there that that you can use in order to take another step. But those are those become the two strongest components of serverless security. So I'd like to take a quick break from just trying to reiterate the same point over and over again about the difference between serverless and traditional applications. And I'd like to go through a story that I think does a really good job at illustrating the importance of permissions with serverless applications. So this is something that happened while we were at pure sex. So this was before the acquisition and we just thought it was a fun story to share. So this guy, Caleb Zima, who I think is with Capital One now, he essentially put up this website called lambda shell.com. It's no longer up. But what he wanted to see is is serverless insecure? Is a tool like Lambda, for example, is it a security risk? Let's find out. So he threw up this website, he put a shell into the application on the website, and he said that if someone is successful with hacking into this application, they'll give them $1,000 bounty. So our CTO at the time, Ori Segel, this is him, saw this and essentially he was like, okay, challenge accepted. So Ori is now with us at Palo Alto Networks, leading a product team. But at the time it was the CTO of pure sex. So we saw that he said, maybe this could be some good PR for us. Let's go ahead and do it. Okay, so the first thing that he wanted to do, you already have access to this shell, right? So from remote code execution, essentially like the holy grail of application security, he thought he was most of the way done already. So the first thing that he tried to do is to get the environment variables and all of this is no longer online. So that's why we have no problem, you know, putting this on a public webinar. But he tried to get the environment variables from the shell in order to try to impersonate the Lambda function. So he exported the secret access key and the access key to his own machine. And then when he did SDS get color identity, essentially, he tried to check how does AWS, how does AWS sees me as the role of this function. He saw that he actually assumed the role of this function. So he effectively impersonated the function itself. So in his mind, he was essentially in like he did it. So right after that, Ori was walking around on the smartest man alive. This is awesome. He was genuinely very, very happy. Afterwards, he tried to playing around with the I'm permissions a little bit. So he tried to list the different functions. He tried to run a few commands. And he saw that he failed miserably. Now, this is interesting because he got access to where the code is, right? He got access to the function. So in any other form of application security, that could have been game over. But because there was a strict I am profile around that lambda function, then he wasn't able to really do anything other than what this function is supposed to do. So there were pretty strict I'm permission. So it's, it's kind of, it kind of reduced the blast radius around that function. If that makes sense. So again, he tried to do a few more commands to see if if he has the permissions and he kept getting these access denied. So at this point, he was pretty bummed out just like moping around the office because, you know, at a startup, you have to move fast and you have to constantly be delivering stuff. It just like breakneck speeds and he was wasting, you know, like a couple of days on it. It was not a good time at the office. Then one of our researchers suggested that there's, there's probably an S3 bucket involved. Just, and typically best practice is usually to use the domain name for the name of the bucket where you host the files for it. So he ran this command head bucket and he saw that he didn't get anything in return. And essentially what this prompted him to think is, is this some sort of quirk of AWS or maybe there is a bucket there. And, and maybe so because he saw that he was not getting an error or like a permission denied. So he tried a few times. He tried to do the same command with just another random domain name. And then he saw that he did get the 404. So this confirmed from him for him that this original bucket does really exist. Okay, so he was able to list the different objects within there. And he saw that he did gain some, some sort of access into this S3 bucket. And his next step was to see, do I actually have right permissions? Can I actually do something with this bucket? So maybe he was a little bit drunk off of the champagne that he had earlier. But his first idea was to delete the index HTML file that he found there, which of course, immediately crashed the website. And again, this is interesting because the function was locked down tight, but the S3 bucket wasn't. And we talked about this a little bit earlier, how you have to take a look at the permissions that wrap around all of the different components of a serverless application. So after this, he immediately got a phone call from from Caleb. And Caleb was at the time, I think it was either Defcon or Red Hat. But he was in Vegas at a conference and he was talking about this and showing the website and we completely crashed it for him. But we were able to take the website down before any of our competitors could, which was fun. But we never did get that thousand dollars. So a key component in that story, one of the main characters there was I am identity and access management. Typically, that's the way it's called across all the different clouds. Most of my examples here were on AWS just because that's where most of this happened. So let's talk a little bit about what you can actually do in order to leverage I am to secure these types of applications. So the challenge with the least privileged approach, which is I think pretty unanimously the best approach you could possibly have with with serverless or with functions, it will essentially mean that you want each function to only be able to do the very least that it that it can or just exactly the task that that it's supposed to do and nothing else. So for example, if you have a function that all it's supposed to do is to put files in a DynamoDB table, then that should be everything that that function is allowed to do. The model around around I am is extremely powerful. It just it's hard to get right. Because what do you do when you have thousands of different functions, right, it gets challenging at scale. So the problem typically is introduced with the human factor and it becomes very tempting to use that star button, right, a wild card in your permissions that essentially will give a function complete permission into whatever it is that that it's interacting with. The issues around functions having more permissions than they should are probably the most common serverless security issues that we're seeing out there. Obviously, serverless is still on the rise. It's still something that is in the middle of adoption. It's not as mainstream as containers just yet. But I mean, with the rate that tools like Google Cloud Functions and Lambda have been growing, there's no doubt that it's on its way. So for example, if all the function actually has to do is to put items in a DynamoDB table, but with the permissions you see here, there's a star. It means that if that function is if that function is compromised, it means that that DynamoDB table, regardless of of its own permissions, will definitely be compromised. So when I do this session live, I like to do a quick quiz at this point and talk about how we how you can kind of name and how you have to know how to name and spell the correct iron permissions. Obviously, we're not going to do that here. But what we typically talk about is how do you spell the exact permission to put item for DynamoDB? Because the permissions when you're talking about what the function can do and what a user can do, they look a little bit different. So you have to make sure that you read documentation and that you use the correct right, the correct form. If you do end up giving your function just a wildcard and the permission, just a total permission to do whatever it needs to do with that DynamoDB table. This is everything that it will be able to do. So it will be able to completely annihilate that DynamoDB table. And I just keep using the same tools for this example, because I feel that if I'll start mentioning millions of different tools, it might just get too confusing. So I'm sticking to Lambda and DynamoDB just because that's the example that we'll be looking at. So you can take you can think about I am permissions kind of like bulkheads in a submarine or on a ship, right? Even if the function itself or whatever service that that I am profile is wrapped around, even if that gets compromised, the blast radius is pretty much contained, exactly like a bulkhead on a submarine, that function can only put more stuff into that DynamoDB table. It's not ideal, but it's better if that function was compromised, and it had just a wildcard in the permissions, it's better than you know, that table being completely compromised. Okay, so a few quick tips for getting I'm permissions right. So like we said, trying to adopt a role per function model where you have a specific role for each function that only gives it least privileged exactly what it needs to be able to do. Think twice before hitting shift eight, the eight here is missing will pop up here in a little bit. Sam managed policies are your friend. There's an example for this as well. And there's an equivalent with serverless framework for this as well. So here's an example of what not to do. Amazon DynamoDB full access just the equivalent of putting a star in that profile. Here's a better thing to do where you only give crowd permissions to one specific table. And here's the even better thing that you can do. Give the give your function just a very specific action that it should be allowed to do for each for each function. There is a free least privileged I'm automatic role generator, which is an open source tool that that you could use. Still available for now. I'll link it a little bit later. But it's still it's still out there for now. So I'll have a link for that at the end of the slide. Alright, so before we wrap up, let's take a look at a quick example. And I think this would be a good way to a good thing to end on. Let's take a look at a sample application that we have built out as an example for how you might carry out such an attack. So we have this CV filtering system this HR automated application that we've built that is completely based on serverless tools. So lambda and other abstracted services from AWS in this case. And essentially what it does a candidate will send an email with a PDF attached to it, a resume that will get intercepted by simple email service and they'll get a receipt back just an email back saying like, Hey, we got your resume thanks. And then that triggers an SNS queue. And sorry, an SNS topic. And then that topic will trigger a lambda function. All that lambda function does is essentially it receives the PDF, it converts it to text, and it stores it in a Dynamo DB table. If you've been following around this presentation, you probably know where this is going. But this is the application that we have here built up. So let's let's take a look at it. So I have this Gmail account open here. Since this application rests on essentially receiving emails with PDFs attached to them. So there's no kind of web portal for me or like web page for me to attack. So first, let's see that the application works as a chip. So in the subject, let's say my CV. This is my CV. And let's go ahead and attach my johnsmith.pdf file. So I'm going to send this over and the application will work as we as we saw it illustrated the the file will get intercepted by simple email service. It will talk to SNS. It will talk to lambda and will do the whole things. And in about 10 seconds, we should get a receipt back. So here it is. Thank you for contacting us. One of our recruiters will contact you shortly. Great. The application is working as expected. Now let's say for the sake of discussion that I did not get the job. And now it has become my sole purpose in life to destroy this company that didn't give me the job. So I want to do a bit of reconnaissance, right? I want to understand a little bit about this, this application. So let's try this. HR email again. And once again, let's say this is my CV. And this time, essentially, if you think about it, I don't really have a lot to work with here, except for the name of the file. So obviously, there's there are a few more steps of reconnaissance that we might be able to do, but we don't have too much time here. So so you'll have to trust me that we fake did a few more steps of reconnaissance. But what I'm trying to see as the attacker here is, can I affect the code? Can I do like a code injection attack into this application? So essentially, will I be able to pass this command into the code of the application? Does it accept user input as is? So I'll attach this. And I'll say this is my CV honest. And let's go ahead and send this again. Now, we're not going to wait 5000 seconds to see that we're not going to get the the alert back, the the receipt back, you're going to have to trust me that we won't, because the function will time out. So once I've done this, and I saw that okay, I do have a path into the application, I can go ahead and start my attack. And the way that I want to start this attack, and this is really interesting because, again, the code rests directly on AWS with this case, because it's lambda behind the scenes. So let's say again, my CV. And here, let's say, how could you? Let's attach another one of these files. And you see that now what I'm trying to do is I'm trying to curl a script from past been into this application. And essentially, what this script will do is it will try to read the application code and send it back to me over email.  We can actually take a take a look at the code in the meantime. So this is that code. This is what I'm trying to do. I'm trying to read the application code. And then I'm trying to send this back to the email address. Let's see if we're getting anything back. Should be getting something in a little bit. And here we see that that we did get first, another receipt. So this is the first receipt that we got and we got another one. And second, we see that we got the function code. Let's see what we got here. So this is the actual code of the function. This is really what I'm after. Let's say that I'm going to exact my revenge in the form of data leakage. I'm going to steal the data from the other candidates. And I'm going to, I don't know, sell it. I don't know what you do with it. So I can see that there are some security practices in here. So for example, I see that they're checking that the file ends with thought PDF, which is okay. But I'm also seeing that they are accepting essentially user input here. So everything after the colon, like it was John colon, and then I curled the script there, is essentially accepted into this code. And I'm able to execute that. So this is what I'm interested in. I can see that there's a DynamoDB table. This is where if all the permissions that this function really had was to just put items in this DynamoDB table, there isn't too much damage that I would have been able to do. Maybe I would have been able to like crash the function or like time it out or do some sort of small like DOS attack. But I wouldn't wouldn't have been able to steal the database, which I will be able to do now because I'm going to send one last email, I swear. And let's just say full on villain mode. And let's attach this final file. And this is just again, another pacepin script that essentially what it does is it reads, it scans the DynamoDB table, and then it will exfiltrate it. We can actually take a look at that script as well. So we can take a look at it here. Essentially what I'm trying to do is I'm trying to scan the DynamoDB table. And then I'm trying I want to exfiltrate that data, dump it back into the email, send it back to me here. And we see that we actually did get it. So I got a DynamoDB dump. This is all the information from in there with all the emails and personal information for lots and lots of candidates. I think the most interesting thing here is that this is not a vulnerability of AWS or Google Cloud or wherever you run the function. This is a vulnerability in the code that that the developer wrote in this case, it's us, but this is something actually pretty common. It's because the code is running directly on the cloud platform and the vulnerability in the code itself that we were able to do this. So if you think about it, in this case, AWS didn't do anything wrong, right? Because the function has permissions to read its own code and to export it and it has the permissions to do. And it had the permissions to read the DynamoDB table. So if you take a look at your CloudWatch logs or CloudTrail or anything really, you're not going to see anything out of the ordinary. So it just illustrates the importance of having type permissions around your functions. And obviously there are tools out there that you can use to take security to the next step. But at the most basic level, that is something that you're able to leverage as well as just being aware of the importance of having strong code. All right, so just the last thing that I want to wrap up on in terms of taking action and maybe getting more familiar with this. So once again, I cannot recommend enough the 12 most critical risks for serverless by the Cloud Security Alliance, great, great white paper, very educational for you to go over, read what are the risks, be aware of them. At least some of them are obviously more advanced, but the first few ones really are the ones that you need to watch out for. You can also take a look at the OASP serverless Goat application. There's a link for that up here. That is essentially just an application that you can deploy, attack, play around with and press your friends at parties with your serverless security skills. And finally, there is a least privileged CLI tool. It's currently still available. I'm not sure if it's going to be there for long. So you might as well grab it. It's not the end all be all of I am security, but it's a tool that you can use in order to generate a baseline for a least privileged I am role. So that's definitely something that that you can take a look at. So with that, I'd like to thank everybody for joining us today. And thank you for the CNCF for having me today. I hope this has been useful. That's awesome. Thank you, Ron, for a great presentation. We still have a few minutes for questions. If you have a question that you'd like to ask, please drop it into the Q&A tab. I'll kick one thing off here. Ron, you mentioned that the function had permission to read and export its own code. Is that common? Or is that unusual? So it's, I mean, you have to remember that unless you you deny something, then the function will be able to do it. So in terms of being able to inspect its own code, yeah, that's going to be the default behavior. And those are going to be the the things that you're going to have to watch out for, either with governing them with IAM profiles or finding other ways to block those types of command injection attacks. Got it. Thanks. Looks like Chris had a question. Yeah, so there's a question. What protection does PureSec offer for serverless in context of Prisma? So I don't think we can get to product heavy here, but just for at a high level, PureSec is now part of Prisma Cloud, the Planetary Security Platform, along with Twistlock and RedLock and the other companies that got acquired recently. All of the capabilities in there are all the capabilities from these companies have already been integrated into this tool. And in terms of serverless security, then you can definitely use it in order to detect and block the command injection attacks, behavioral protection for your functions to prevent it from running processes it's not supposed to, as well as getting a very high level of visibility into the permissions of your functions. So a lot of cool stuff that you can do in there for that. But we have a lot of good resources for that online. So we have another question. How does the security policy get stored and applied? Is it via S3 bucket? So the security policy for individual functions, if we're talking specifically about the native tools that you have out there with, for example, with AWS, it will typically be an IAM profile that you assign to the function. So you don't necessarily have to store it in an S3 bucket. If you're talking about other tools like Prisma Cloud or typically or other security tools, then it will typically be some sort of wrapper around your code. So when your function, when AWS invokes your function, so instead of calling your handler, it might call our handler, so we'll be able to inspect the event data. So that will typically be part of it. And the other part will be an authentication with the AWS account to be able to do vulnerability scans and things like that. If I can give any thought, another question is, can you give us any thoughts about security as it relates to alternative serverless systems like OpenFast? I think, so personally, I don't have too much experience with with OpenFast, but I think that it's really different once you're talking about something that that you're essentially deploying, I don't know if on-prem is the right word. The security of the underlying infrastructure is still going to be something that's going to be your responsibility in that case. You're still going to have to worry about the same types of things. If you're, for example, just handing these functions to developers, you're still going to have to worry about the strength of your code. And I'm not sure exactly how permission governance works within OpenFast, but I would imagine that that is still very much something that you'd have to look out for. But I would say that the biggest difference is going to be that you're still going to have to worry about what sits underneath while with public cloud, which again is just where I have some personally more experience with serverless tools. That is something that is just not even on your radar. So for example, the the networking or the operating system, like those things are there, but obviously they're not on your radar. I hope that makes sense. Okay. Any more questions? I think that about wraps it up. So great. Thank you, Ron, for a great presentation. Thank you everyone for joining us today. The webinar recording and the slides will be online later today. We look forward to seeing you again at a future CNCF webinar. Have a nice day. Goodbye. Thanks, everybody. Bye.


All right before I get going Who here has heard of salt stack before? Okay, most who here have deployed salt before So no one in here is managing a production deployment of salt. I know I know you're involved in one Okay all right Now The primary point of this presentation is to go over how a lot of these large Web scale companies are using salt and talk specifically about a number of the points That are in there and tied into a number of other technologies that are out there and talk about how they come together to create complete infrastructures Now the problem here that we face is that this is being cut off on the left One of the problems here that we face in modern infrastructures is the fact that we've been blasting around these AAS Phrases like mad over the last few years and the problem is is that we need to move well beyond what they Offer what we need to do is start talking about or talk about more What goes on top of those platforms? Not just how to build them and so a lot of what I'm going to be going over today Has to do with a number of different infrastructures how they work and Well, how they're built Because what it boils down to is that there's a lot of infrastructures out there that are built on infrastructures the service clouds or Platform as a service clouds, but there are a number of hardware infrastructures, which are very Web scale or whatever you want to call it now I'm going to go over just a few examples In here that have to do with a couple of cloud-like companies And so the first one is going to do with how wikimedia Uses salt and a number of other tools for continuous deployment of code and the continuous deployment of Wikipedia itself and Then I'm going to talk a little bit about some of the techniques used by LinkedIn. They are a bare metal shop But they have just shy of 30,000 servers right now and Then lastly I'm going to talk about something that's a little more next generation salt stack the The company behind salt we recently got a contract working on some really interesting virtual machine auto scaling applications and I'd like to talk a little bit about What we're doing with regard to that and also some of the new stuff that we are in the process of Developing and delivering and of course making open source so If we start up here I'm going to start by talking about the problem of continuous code delivery Now there are a lot of ways to tackle the problem of continuous code delivery as I imagine many of you are well aware but Fundamentally, there's a number of core things that need to be considered now among those is that that continuous code delivery mechanism needs to be directly tied into some sort of continuous integration system or Directly tied into revision control systems so that it can be delivered continuously The next is distribution of code out to different locations so that you can have different types of file distribution and fan out mechanisms Service interruption is very important obviously because what you end up looking at is that for extremely large websites and very professional deployments They like to go through great lengths to make sure that you never hit the website During one of these code deployments because they could be very frequent You can have many code deployments in a day You don't want users to hit that website in the middle of a deployment and something wrong to happen Because files were in an intermittent state. So we'll talk a little bit about that and then of course the idea of taking your tools or your mindsets or your Methodologies and turning them into a culture inside of an organization Which is where a lot of these concepts of DevOps for instance really come into their own because it has to do more with communication inside of an organization So I really like the Wikimedia solution for continuous code deployment the person who put this together Ryan Lane very intelligent fellow so I'm going to do a quick overview here and then come back and readdress some of these issues So we start out By the fact that Wikimedia is using Jenkins for testing of their code bases And so they've got tests and builds being executed on Jenkins, which is probably something extremely common in this room Who here is using Jenkins? Okay, okay, that's 62% Now what they do is if we look at a salt topology salt is made So that you have a master or as we were discussing earlier you can have multiple masters and Then those masters are controlling minions Now all of those minions are able to either communicate back up to that master or receive information down from it Now and by information, I mean they can receive instructions or they can download code or Get all everything that they need for say configuration management routines that are highly complex now in the jet in the Wikimedia situation what they're doing is that they have salt master and Minions and I don't want their minions sits the Jenkins server. Okay Every time that Jenkins server Executes or finishes a build which has been tagged in such a way to invoke a code deployment They use salt pier system now. This is the ability for Minions to go send a message up to the master and then request that somebody else in the infrastructure execute on said message and Then return the information back to that minion So the pier system allows them to have a plug into Jenkins that fires off a command to the salt master to say Hey, let the web tier know That we have fresh code for them that information then subsequently goes down to the web tier Tells them. Hey update your code They check into Jenkins. This is of course within a second of that build completing Download that information and deploy it deploy the code fairly simple and straightforward now the next cool thing here is that They want that information about that code deployment To be living somewhere outside of this simple command structure Okay, they want it inside of a redis database that's being used By their internal web front end that tracks everything that's going on at Wikimedia on the insides and So salt has a solution for that called returners All of these commands that we talked about in salt where we have a central unit that splays out lots of instructions Down to its minions happens Asynchronously now the benefit here is that we don't really care about necessarily that message coming directly back up to the source We can redirect the return information to any arbitrary Location the most more commonly that ends up being a database. So in this case Wikimedia uses returners Inside of the salt minion to say once you're done deploying this code Go load the information about this code deployment up into a redis database And then that is instantly available to their internal Their internal web interface So at that point they're able to see that deployment process from start to finish and Have all of that information presented for them and Nobody has to touch the infrastructure to make it happen. It is completely and fully automated all they have to do is Set some tag and get the Jenkins is watching for Okay Now if we move a little forward here Wikimedia was the first really big scale Company to deploy this sort of setup Since then we've seen it done a number of different ways and we've worked with companies to do it a number of different ways And since then a substantial amount of additions have been placed into salt so Again, we started from that original source of a build server Just got build bot and bamboo with some other options up there Trying to think of some other Options those are the ones I've used the most heavily personally But so as soon as those builds complete instead of using this peer system I was talking about since Wikimedia set up their deployment We've added something called the event bus inside of salt now this idea is that Everything in salt is events. It's all entirely event driven And so whenever any action completes events get fired in specific locations so when Jenkins finishes a build it fires an event on salt and Tells on the minion and says hey tell the master that this build event has completed This is very simple to do because salt's event system is open-ended Any external application that has sufficient privileges on that machine? So generally root access on the machine, but privileges can of course be granted in other ways Is able to fire its own events on salt bus? So that event then goes back up to the salt master where we have the reactor now the reactor is something that We're going to talk about a more depth a little later But basically the reactor is listening to these events as they fly by and It's able to subsequently make a decision and say given set event. I'm going to automatically execute a Specific routine and so We are able to configure the reactor to just say if I'm given a deploy event I'm going to fire off a deployment to the systems to which that deploy event has requested a deploy event, okay? Now similarly, there is this problem of how we're going to be distributing files Now when you get into really big infrastructures, this becomes a problem in smaller infrastructure say sub a thousand servers It really isn't that big of a deal. It can be if the files are quite large of course But not as not as regularly as when you're dealing with say Twitter sized infrastructures and so When it comes to code deployment and update Salt has been built with an asynchronous queuing file server now the idea behind this is that by using AMQP Techniques, we've been able to build a system that cues multiple connections back to salt's own file server So that it is optimized for the distribution of files across massive numbers of systems in parallel Now with that we see many infrastructures having many many thousands of servers requesting files from the salt file server simultaneously and It's able to keep up with that and subsequently deliver those files Also salt has the ability to hook directly into package management systems So if your Jenkins system is building Distribution packages if that's the route that you've taken The networks same with plugging into systems like pip or c-pan or Gems or what's the one for a heckle? That's PHP one yeah so we've got a number of systems that allow for Pulling down and managing connections and code deployments when they're packaged as well as getting them directly out of version control software now the last thing Is the question of where the results go for that continuous code deployment? Now you've got a number of options in this one is that we recently released open source, of course the Heylight web interface, which is the UI for salt. It's a little new It's a little raw still needs some features all of it But it is coming up quite quickly and so the information from those code deployments can be displayed in Heylight Or you can do it the Wikimedia way send it to a database and have some other System view it now this returner concept. I was talking about earlier the idea that the salt minion can go in and attach to any external database or interface Allows you to easily configure messages to say that when this message is done Send an email to somebody or when this sorry when this routine is done Populated database. Let's see send an email send an SMS Post information about the successful build on a specific chat server. All of these hooks are already in there Making it again really easy to just say redirect that data to wherever Okay. All right Any questions comments? Yes Down Okay, the question is if if anything is down in that chain of these questions delivered reliably now Inside of salt if a master is down then The minion who is shooting an event up to it holds that event in a queue and Waits for the master to reconnect when the master comes back online, then it fires Okay If the master is sending out a publication or a command down to minions and minions aren't online Those get discarded primarily because you don't want a minion that's been on off-line for two weeks to boot back up And then replay a bunch of old messages So to answer your question Uh-huh. Yes Correct With that said we do have a number of routines that you can that that can be easily configured on those minions To tell them to come back up to speed when they boot up. Okay So one of the next problems that I want to talk about is the classic case of Extremely large infrastructures So extremely large infrastructures is something that we're seeing more and more Pop up and a lot of salt users just even so users who are using us on very small scale really like the fact that We have these sorts of deployments that we work with on a regular basis So they know that if they need to scale and many of them hope they'll need to scale in the future I mean, especially if you're a startup you always want to need to scale in the future then they're assured that the Deployment and system that they're working on is something that can grow with their company and with their deployments And so when we have to start looking at Communicating with many tens of thousands of servers and managing tens of thousands of servers some new problems prop up and so one of my favorites from LinkedIn is About close to maybe a year and a half ago There was the Java leap second bug if you recall leap second. Well, it's a timing bug in Java They were sitting there and they had a meeting Discussing okay, what are we gonna have to do to deal with this? We're gonna have to log into these servers We're gonna have to verify that they're up to date. Are we gonna build shell scripts? They're gonna do it automatically. Do we have SSH systems that can handle this? And they were only a few minutes through the meeting when a guy in the back raises hand and said We're done I run a couple of salt commands We're done. We fixed everything in a few seconds and so Salt itself is made to start having these tools that become very apparent once you get into larger Deployments now. I've got on my laptop here. I am simulating Let's see if Here we go. I Am simulating a somewhat no, it's not a very large deployment because it's just a laptop But what it looks like with salt to have 25 servers attached and so when we run through Adjust this I want to get it cut off All right So we run through and look at what some of the basic functionality of salt is it really rotates around this element of remote execution and The idea of sending out perfectly parallel commands across large volumes of systems and so if we wanted to Say get the network information of all of our systems. We're able to get that information in just a few seconds Another nice thing is that salt still keeps going quite quickly as it scales So one of one of my favorite quotes was from Harvard University when they deployed salt on a 1800 node High-performance computer and they said that the old method that they used Which used the parallel SSH tool to fan out and execute commands on their entire? Supercomputer cluster and then return that information back took them roughly 15 minutes to run Okay, and then they installed salt and now takes them less than five seconds So it becomes very useful. It really opens up a whole new world of what you can do because all of a sudden you have real-time stat gathering You've got real-time modification You can go out and say oh well. We've got a disgruntled employee We need to get rid of his password right now and you can give it a business H keys everywhere Or similarly you can say we've got a security hole that we need to fix in our entire infrastructure right now And you can run that routine that's going to update those packages Everywhere at the same time Similarly it may it's very easy to allow it to stagger those updates So you can say okay update five percent at the at a time and a roll through the entire infrastructure's deployment So when we come back and we start talking about larger infrastructures This is something that becomes Well very important Okay so LinkedIn started using salt and frighteningly early. They were using it in 2011 Before I had deployed it in a real environment It almost makes me regret start, you know, that whole first commit being open thing People were using it But they recently came back and rewrote all of their salt deployments to take advantage of a lot of the new features But some of the main things That I was talking about is really central to how they manage their infrastructure today a Lot of what they do has a lot of what they do today has to do with the ability to gather real-time statistics About what's going on It's their ability to go out and see exactly what version of software Exists on all of their systems and then find those systems that have differences and get that information reported back in a few seconds and Subsequently go out and make modifications So that it's very easy to run a couple of salt commands and then ensure that every bit of code across their infrastructure Of just shy of 30,000 pieces of bare metal is consistent Okay Next we talk about triggering code deployments now They don't have a continuous code deployment that works quite the same way as the ones which I've already shown But they do illustrate a point that's inside of salt. That's really cool So they have code deployment processes that are very very tightly tuned So that they're rolling through servers in a very clean way Now what they do is that they want to make sure that the load balancer has taken a web server out of the mix before they start modifying the code that's on that web server and So salt comes with the system and it called the pre-wrest which was actually designed with LinkedIn's assistance, but it's quite slick So it allows you to define inside of your configuration management systems in salt Which now I'm regretting that I'm not talking about more in detail. I just I like talking about it but it allows you to define something that says Only run this routine if we expect Something to be modified in the future something specific so you're able to define and say Go ahead and tell the load balancer to take me out of this system or shut down the web server Only if I'm about to do a code deployment and Then once the code deployment is done restore that previous state Which means that you can maintain Very complex item potency concepts inside of inside of a deployment Okay It also means that they can fully automate that code deployment process in such a way that again Completely minifies or they hope Nullifies the possibility of continuous code deployment causing any hiccups in service and reliability Now the last thing that I want to mention With respect to LinkedIn's deployments is a concept inside of salt called runners Now one of the problems one of the complaints that we get about salt is That we have lots of terms for things And we find people come back and say oh you've got all these terms for things And we reply and say well that's because we have lots of things that salt can do Now you don't need to learn all the terms in salt You can be up and running hardly knowing anything at all after reading two pages of documentation You'll know how to start using salt for more execution and config management Actually, I think that tutorials like two and a half pages. It's not very big but This next concept of runners is the idea where you're able to set up complex orchestration routines on a salt master that can Deploy systems in complex ways directly from code So you write in Python a runner that is able to hook into the salt system and say All right, we're going to start by executing states The config management system in salt on these servers We're going to stop at certain phases and many of these tools and features that allow you to control this from Python are built into the system and Then the runner is also able to aggregate data returns That are coming back from all these minions make make decisions and then fire events back off to this reactor system So again, you're able to start making very intelligent Automation routines if you need to a very small subset of our users actually have to go this far And they usually only ends up with very complex deployments, but The tools are there a lot of the idea is that you don't need to deal with all of the tools in your toolbox But if you just happen to be in a situation where you need them We don't want that toolbox to be missing something critical okay, and questions comments Laughter like my jokes are just I don't have any today. I'm not doing very well there Okay Now the last thing that I want to talk about What do we started we've got till 540 so okay 20 more minutes The last thing that I want to talk about it starts to bring a lot of these concepts together And this is where we start to talk about more autonomous computing systems Now I mentioned this reactor system the reactor system insult is pretty straightforward It listens to these events better being fired if we have time I'll show you what these events look like and show you a little bit more of the salt command line and Then reacts to those events. It's fairly static. It's very two-dimensional in that regard It gets an event and it does something with it now one of the problems that we've that we've been facing quite recently is That we got a company that wanted to do virtual machine auto scaling, but they wanted a level of flexibility That wasn't available in other tools And so they came to us and asked if this was something which could be created So we went out to some of our contacts who had been building Autonomous submarines for the Navy Because I knew that they were sitting on some Python code That would that to my happiness will work very well assault. It's entirely event-driven That they had been licensed to be for them to open source And so what's going on is that we now have and are bringing in an open sourcing. It's called Ioflo a Logic engine that's used in robotics that is configured by Very simple data structures, so you write some YAML and it configures a robotics engine The entire YAML needed for a submarine mission was generally less than 40 lines So it's extremely terse and extremely Direct and allows us to do all the levels of tree logic and aggregate and and state aggregation That we need to to be able to make highly complex decisions about an infrastructure and then automate forward routines so This new system is something that we're actively developing in response to this virtual machine auto scaling problem and In response to the ideas or concepts behind intelligent automation inside of a data center And so what's going on here is that now we are getting to the point where we can aggregate events across a tier So that it's very easy for us to get all of the events relative to the performance of say a web tier and a database tier and Whatever else it is that's in your infrastructure say your AMQP servers that are passing events around Take that information back and it creates a state inside of this logic engine and Then when a certain state is reached It fires an event off to the reactor to say We're in a bad state of this type and a nice thing here is that it can start making extremely complicated Decisions about what's going on in an infrastructure because we have underlying access to all of the components and a completely free-form logic engine And so we're able to make decisions that say all right So the load is going up on our web tier great If the load is going up on our web tier then what we want to do is see what the disk IO load is on Our web tier and on our databases so that we can determine if this is going up because of high IO weight Or if this is going up because of too many connections and start to make much more informed decisions about what what's going on And then allow that logic engine to do things like say well given our current patterns We're going to go ahead and start deploying virtual machines to the web tier which are of a different shape and size We are able to communicate back to the database to say we're going to expand There's no SQL database to be able to handle this higher load Or we're going to be able to communicate that we want to Build a fail-over-rabbit MQ server that has higher load and then migrate the load over or higher capacity and migrate load over to it And so these complex routines is primarily what we're working towards right now now I Just talked about all those now the simple solution Today is to build that logic into a runner and Python That's going out and pinging the environment every couple of every n seconds Or minutes or whatever you want your Window to be and then making decisions internally But of course we wanted we want something that's substantially more powerful And I'm really excited to say that the IO flow libraries that I'm talking about We did get those open sourced earlier That was last week So those are up and available on github now there. It's really cool stuff Okay, so does anybody have any questions or should I dive into a little bit of a demonstration? Yes Okay, so so the question was in puppet you can have configuration files that are replaced or updated from the master Okay, so salt is able to do all of everything config management wise that any of the other guys can do so yes you can Maintain configuration on the salt master and then communicate that configuration back out to the environment So it really is like M-collective plus puppet And it's entirety together There's a number of There's a number of things worth worth pointing out when we start looking at salt's configuration management systems So I'm just going I've only got some simple ones on here. I wasn't expecting to necessarily dive in here But salt's configuration management system is all based on YAML Or rather I should say it's not based on YAML. It's based on data structures And so all the matters is that we get data structures into the system And so this is a very simple example where we're installing Apache We're making sure that the Apache package is being installed. We're deploying a Configuration file to the wrong location. I know I wrote this quickly What is it at the HTTPD slash cough Dot D slash HTTPD dot com Don't do it every day anymore And then make sure that that service is running so again a very flat simple example now if you want to have Turing logic and advanced variable scoping in there. You can use ginger templating Now that means that you can do for loops in there. It means that you can do If else statements and it has full access directly while rendering these Files into data to execute any of the routines on the command line Inside of the remote execution system on the system that it's building on so that you can very easily shell out or Run built-in routines take that data back and then munch it into your into your system. Okay The other thing that I want to mention is that one of the big differences with salt is that you might you may have noticed that we've got These were these statements like require and watch Now this is a more puppet like thing Where we're ordering who's going to do what when and also what they're depend and building dependency chains And so we're able to do this again very simply by saying that Well, this package is going to be required by the HTTPD service down here The HTTP service is watching this file. So if this file changes Then you're going to restart or reload the service those sorts of constructs. They're all in there now the example I gave of reading into the future. Are we about to deploy code and Making a decision about something which is going to happen is another requisite called pre-rec So we pre-require that something else is going to be modified Now the next nice thing though is that In the absence of requisites or rather I mean with with requisites in there But it's a little crisper with the absence of requisites everything in salt is Executed in the order in which it's defined Which means that these runs will always execute in the exact same order and they will always execute in a highly predictable Indictable and understandable order because they're evaluated from top to bottom in these files And similarly, it's really easy to tie all of these files together By just using that include statement up there Where you come back and you say well, I'm going to install say mod SSL I'm going to include Apache. We're going to make sure that Apache now requires mod SSL when I'm using this this formula as well And similarly you can extend Earlier scopes and systems so you can build hierarchical components So all of all the components are there. I mean without me Blathering on about it for another 20 minutes or well another two days Yeah, all the components are there and the goal of course being to make it simple extremely predictable high-performing and Entirely featureful. I'm sorry. That probably is far more than you are asking from your question instead of me going Yeah, yeah, I can do that Okay, any other questions are you tired have I not been very entertaining Okay, so part of your setup process requires user intervention Okay, okay, you've got a couple of options here One of them is that we've got quite a few salt modules that specifically take care of those interactive types of scenarios So we've got a lot of heavy wrappers around SSH routines We have heavy raptors around generating certificates and keys to make that easier We also have a system inside of salt That makes it very easy to have a central key signing authority or certificate authority on the salt master and have salt generate keys Sign them and securely distribute them for you So I guess in a nutshell We've covered many cases But we don't have say a generic way to work with continued Standard input and output inside of commands in a YAML like fashion We have all those tools code-wise to make it very easy to build those routines built directly into salt Okay That answer your question, okay All right, then I'm gonna cover a couple of concepts really quickly since we've got a few minutes left and You're You're not asking as many questions as I'm generally used to Probably is my fault. I'm sorry. Okay Now inside the salt system. I Showed you whoop Executing a simple command now this test.ping command Again is just sending out a command that says ping to all of these systems and all it does is return true very simple But salt ships with a very substantial library and is self-documenting So we can run salt.sys.doc. We'll pipe this guy over to less here and now We can come through and take a look at All of these different routines and they all have examples on using them Built directly into the remote execution system and so let's see we've had some ACL stuff Apache archive Routines working with network bridges There's more ways to shell out than you can shake a stick at I'm trying to think if there's a Scottish phrase that phrase. I know for that, but I don't okay Let me keep going CP allows you to interface with salt file server directly instead of the fact that it doesn't does all that stuff Automatically inside of the config management system, which is another concept. I should point out inside of salt everything is built in layers Now this idea being that any component down the stack that you want to know how it works Or maybe you have a legitimate need where you need to interact with something lower on the stack of Operations everything is exposed all of the routines that are executed by the configuration management systems inside of salt Are all exposed inside of the remote execution underneath the hood? So if you're wondering exactly how Packages are installed. Sorry, that's how files are managed through there If you're wondering exactly how packages are installed you can come down here and Find the package module which leads me into another point about how this system works It does automatic normalization on every layer between distributions and what their underlying routines are So if we come down here to package There it is All of these package routines are going to smartly know to use The routines from the underlying package manager on the system that they're targeting so you can send a command down it'll know to use yum or Pac-man it's arch Linux or apt-get or brew or Chocolaty on Windows Or anything the other nice thing is that these commands not only Execute on the right systems on the on the target or with the right software on the target systems But they normalize all the data return that comes back Which means that you can execute a single command to say list all of the packages on all of my systems and All of that package data is going to come back in a day in a data structure, which is completely predictable Okay So that again, it's the raw and important data gets normalized now looking at this one more time with data being Normalized and managed if I do something like Package that list packages will see that almost instantly I go out and I gather that live package data About the target systems and it comes back in a data structure Now this is made to be printed as in a very pretty way by default But everything inside of salt is JSON serializable the idea there being that Again, you can take all of this raw data and shove it into anything or Shoves raw data from anything into the salt workflow This is something that comes up a lot especially with these use cases. I've been talking about Because we talked about integrating with Jenkins It's really easy to send data out of salt and translate it into something that Jenkins knows because salt's always just gonna speak Jason Okay but not only that we can do this and Tell it to output that information to the display in Rajasin or in any other format that we would like so Yammel for instance You know, we don't have one for XML because no one's ever wanted one If someone does it will be made But it hasn't happened yet And I make this joke every time and still no one's made one just out of spite or something All right So the last thing that I want to mention inside I'm in here and then we'll be all wrapped up Is a concept inside of salt called targeting So one of the major problems we run into when managing an infrastructure is the question of who is doing what? It seems like a very simple and straightforward problem But for those of you who have deployed especially clouds now because you're just spinning up virtual machines all over the place You realize that that can turn into a bit of a problem Now many existing systems are fairly static in the way they make those determinations They come back and they say okay, we're gonna go by host name You're gonna name everything according to a certain convention or they're going to say that we're going to store this who is what? Construct in some database somewhere Now salt leaves it very open-ended for you to choose how to go about doing that And so the way that that works Is that I've been using just globs here To specify that we're going to be talking to everybody But similarly we can use I know this is contrived I'm not exactly a reg X master, but we can use regular expressions instead of glob matches Which appears I may have found a bug in the development version of salt Or we can do something called grains Grains are bits of static Information about a system that gets generated when the system starts up If you need information about a system that isn't static information Then you run an execution to gather that information because you need to execute a fresh routine to gather it Right if it's static we can store it in memory and we have extremely fast access to it Salt is be built from the ground up to be high performing But similarly we can match systems based on these grains and we can statically set them So we can have a grain and say tell this minion that its role is A web server In its grains and then We can say all of the systems with a grain of Role equals web server are going to do these things Okay And so you're able to do hostname validation Or information validation similarly you can target systems based on their network information Or you can use External databases to gather information about systems if you want to Or you can use what we call compound matching So you can say I want to target all the systems That are running Debian But not the 32 bit ones And only the ones that are in this subnet But not The ones that are running on Lenny And put all that together and then it's going to Just give you the ones that you want So very very fluid and open in the way in which things work All right, I am out of time And I'm I'm sorry. Usually I can get some chuckles out of people But I must be like jet lagged or something But yeah, I'm out of time. Does anybody have any questions before I let you go? Any comments any rotten fruit? Any rude Scottish remarks? Wow, I didn't get one Yes Sure Okay, yeah, and I mean this stuff is all High level because yeah, the reality is is that I mean, well, there's reasons why we have training classes I think there's a lot in there And a lot of this is yeah, just to give an intro But yeah, we can do that Okay, any other sorry any other questions comments, arguments, rebuttals All right I'll let you go then


Hi, I am Dr. Yashaka Gudey sir. I am consultant and consultant at Manipal Hospital, Dwarka. I have an experience of 18 years. Genital hygiene is something which is not talked about and which is a very important topic for every female. Most of our educated young children, they need to know about genital hygiene. The right age for genital hygiene, actually I will say there is no age to learn. But if the age has to be taken, which age a child should know about it, that is the adolescent time. Though a child watches her mother and other people in the family from a very young age, but the adolescent is the right age when we should tell our children how to take care of the genital hygiene. So, genital hygiene in our country is coming down to our children, to our elders, to us, from what our mothers and our grandmothers have taught us. But there is a difference. Some things have changed, new things have come. Like we should realize that excess of genital hygiene will lead to a lot of infection. People who have diabetes, people who have antibiotics and who clean themselves, who have a cleanliness freak, have more chances of genital infections. So, the cleanliness of the genital area should be restricted to once or twice and not with harsh soaps or any other chemicals or did all. It should be just a lactobacillus wash, simple water, simple lactobacillus wash once in a day. That's enough for the genital hygiene. Keeping your genital area wet every time washing leads to lots of infection and lots of probiotics. We should increase the body immunity so that we don't have infections again and again. And any issues related to genital hygiene, all the adolescent children I think at the time of their puberty should meet our gynecologist once to know how they have to keep the genital hygiene at the time of their puberty, their periods, what are the other methods available here. There are so many other methods available like menstrual cups, tampons, sanitary pads which are of different materials. We have deoxyn-free materials which prevent from cancers because genital hygiene is very important to prevent our younger generation to have cancers in the future. So cancer will not develop at the age of puberty or at the age of reproductive age. It will develop at the age of maybe at the perimenopausal age. So starting it right at the childhood at the adolescent age will be the first step to prevent any cancers in the family.