to choose which of them because they have different identifier with MIT the situation is even worse because MIT in Callaway system represented I think eight or 10 SPDX licenses and it may hide even some licenses which don't have the SPDX identifier which you have to apply for and it may take a week or two months to actually get it from SPDX and get it to federal license data so if you want to work on that you should probably start quite as soonish. This is all packages in Fedora like packages in Fedora doesn't have this SBOM related to as far as I know in Fedora we generate the SBOM for the containers where we take the container as only one part and then one file so as far as I know we don't dive into the container itself so it's super easy bill of materials. Yep, so the question is why the trivial conversion are not really converted by automation? Because I'm in a group of four, two lawyers and I'm one of two engineers there and all other things that we should evaluate it manually because we at the same time change the meaning of the license so previously we can evaluate it, the license now we should not evaluate it so the license string in some cases actually change even if it would be conversion from Colovay system to Colovay system because you add more licenses with some operator license one or license two for example and sometimes the license evaluation was done pretty long time ago and it may not be true and it actually happened, it's not so rare like last week I was trying to convert the RPM itself the RPM package and it was not straightforward like RPM web set this GPL version two license the copying say this is GPL version two with some exceptions for RPM IO and RPM leap and license string in RPM header said something different so we have three cases and the issue is still open in RPM GitHub and I'm discussing with Panum, Miro Harenchok, Neil like what's actually the final concluded license so it's not straightforward even in those trivial cases so it's hard, trivial is not so trivial so the question is whether there is problem that we have two standards and whether we can convert from one to the other I don't know, I like this but I don't know I never work with this format directly so as I mentioned I'm coming from the license part so I was just screwing what's around me around the license part so I don't know so the question is can you ask our federal package maintainer help? Yes, yes, yes, yes you can and please do because you know your package well and you know like whether there was some old system like the RPM which is 20 year old even more or whether it's new and like what the upstream say is true or you know which files are used there and whether it's really trivial and you can convert it from GPL v2 to GPL dash version to dash 2 and that's it and you have five second job or whether you audit it and know which files are there what can be the problematic part so you know it very well and we have tools like license dash Federa 2 SPDX which can help you convert the strings there are other tools like scan code, Ascalon of CLI, license check which can help you audit the files so this is where you can help and we are organizing workshops so if you hesitate with something we can help you but you know more than we know so please help us. And the ultimate goal where we are doing that in Federa converting like the material is things SPDX licensing strings are other things and any other future software which builds on auditing on management of licensing will build on some injustice standard and that seems to be the SPDX especially about the licensing so we don't want to do something for just a moment which will handle the Tom Callaway system so we want to use something new and injustice standard so other tools somebody else will use use the new standard things and we are there as well as Federa first other question okay so no other question if you will have any later find me if you want to help with conversion of your package to SPDX identifier let me know and I will do my best so thank you Are we live? Are we live? Yes, you're live. Perfect. Welcome everyone and first thanks for the DEF CON sponsors to get us here today I'm here with my colleague Alessandro we are part of Red Hat side reliability engineering team working on managed services with a twist which we come to later and our talk is about how we manage to retain some of our sanity in what we do and show what we've built over the last past few months so Alessandro please Okay, so my name is Alessandro I'm a senior SRE at Red Hat and let's start and let's start with our talk with an example application that will help us through the whole talk we have the three standard layers like UI API and DB again here we have some an example of the resources we may deploy on a Kubernetes cluster but this is just an example there should be more so let's focus on the three green one so two deployments for UI and API and one for the DB it will be helpful later and let's assume you already have this beautiful application running on your Kubernetes cluster and everything works fine life is good but then at some point the dev team wants to deploy an update for your application right so you as an SRE you don't have already big processes in place so you're just started and you get from the dev team you get the good old folder with some YAML files containing the deployment and the state full set for the database you look at it it's all fine so you accept the challenge and you deploy it so what do you do you use your beloved kubectl command you apply the whole folder and after a few seconds everything seems fine so you're like very happy about how cool Kubernetes is such what a beautiful tool you're using but then you're a smart guy so you go and check did this actually work so what do you do you go and check the deployment and the state full set using your beloved kubectl and you see that everything is running fine so now you're super happy and you go grab your coffee but then a few moments later when you go back to your laptop you look at your slack icon and well it looks like this and you go and check one of the chats and you have your annoying dev that tells you hey man the new UI is down so what have you done and then panic starts and you go and look and what the hell I mean the pods for the UI are all crash looping what's going on so panic keeps going but then you have a moment of sanity and you remind yourself that you can undo what you just did right so you regain your call and you just undo you just basically roll back the deployment for the UI and you think that everything is okay but then after a few seconds then you're panic again because it's still crash looping so what the hell is going on and you write to the same dev in the Slack chat and the dev tells you well the ODI don't work with the new API because we make some breaking changes there so you're still panicking but you think I can think about doing it again so roll back also the API deployment hoping that the database have no changes right you try to do that with the same command before and it rolls back actually rolls back but at this point you regain your calmness because you see that everything is running fine so luckily the database had no changes so rolling back everything solved the problem temporarily this is a typical day in the life of an SRE so things keeps breaking randomly and you don't know why but let's try to see what happened the events that brought us here so we received some YAML file from our dev team we trusted them and then we used our cube-cattle tool but in the end we could have also used Customize or Helm to do the same thing and those tools, what they did they just pushed resources to Kubernetes and let Kubernetes do the reconciliation I mean I said just because that's what they are designed for so they worked as expected and now I have a question for you which is which step is to blame for the failed update? The developer? Yeah The YAML you trusted Exactly well no the YAML were fine but things keep breaking for some reason so we don't know why the content of the container may have been broken but the reality is that in this picture none of these steps are to blame for the failed update so we actually have a problem from an SRE point of view because our tools are like well I mean we did our job so don't look at us for the failed update okay it's not our fault but in the end as SRE again the update failed right and the update could have caused our SLOs to fail as well and the problem is that the SLOs are the actually actually I think that matters in the end for an SRE right so we have a problem but we as an SRE our mindset is to analyze the problems that we face and try to turn them into ideas to improve our workflow our tooling blah blah blah so let's try to analyze the problem we just had first of all we have loosely coupled objects every object that we push to the Kubernetes container is loosely coupled so what if instead we have something that allow us to bundle those resources together in one package? Spoiler second point is the state to solve all these coupled objects are distributed and heterogeneous so it's hard to understand it's hard it takes time to see if the deployment succeeded and all the other resources so what if instead we had an aggregated status for this bundle of objects that tells me if the update actually worked right the third one is all at once roll out so you push all the resources together at the same time and then you let Kubernetes do the reconciliation but sometimes this is not enough so what if instead we can have we could have an incremental control roll out of those resources right so for example we try to deploy the DB if it works then we deploy the API and if it works we deploy the UI right and the fourth and last point for this slide is complex rollbacks because we saw that the deployments have an easy mechanism for rollbacks right but what if we could have the same concept for any type of resources that we deploy to the Kubernetes cluster that would be very nice right and now I'll hand over to Nico for the second part yeah so I'm now here to tell you why we can't have nice things in what we do because of scale and compliance right scale yeah everyone on scale and compliance boo right and if we take the ideas that Alessandro already introduced and tried to turn into solutions we had some excellent talks during the last days talking about things like hey we need to bundle our resources together right so we want something like GitOps right you put all your stuff into a repository and if you need to roll something back you change something in the repository and magic happens and it appears in production if you have a problem with status right you don't know what's going on you have monitoring at least I hope you have monitoring right but there are solutions for this right and even with four more complex processes like incremental rollouts you can rely on ArgoCity or essentially Piggy or Poison right there are plenty CICT tools out there who have plenty instructions on how to set up sensible rollout strategies and in the end for rollbacks right you can make backups Valera is a nice tool for Kubernetes backups or there was a nice talk here a few days ago about Argo rollouts which also includes automatic rollbacks super nice so can we have nice things well yes but actually no so all of these projects are super nice I highly recommend them but I can't use them and well first because of the scale that we're operating at my team we are not operating with like tens of clusters or a few hundred clusters we are tasked with operating across thousands of clusters and closing in like ten thousands of clusters admittedly not all of them have managed services deployed that we take care of but still all of these clusters need to have the ability to install managed services at any given point in time right because in the end our customers they don't care how we make the magic happen in the end they just want their push button deployments right they swipe their credit card get their quota and then they start installing stuff and they don't care that our backend infrastructure has a hard time coping with like thousands of clusters but this is where stuff becomes tricky because all of the tools that are commonly available they don't quite work at this scale and then there's an even bigger problem those are not our clusters those clusters belong to our customers they run in their AWS their cloud accounts right so just music stops because this is a big one this means well one does not simply send data out of the cluster right we can't just take whatever data we want and ship it outside of the cluster because even such simple things as the name of namespaces in the Kubernetes cluster right it's just like a folder name in kube they might contain sensible information about a customer's next big project right you can't just take that and put it into our management systems we also don't want to install random open source projects there like our CD is not random we can't install it on the customer cluster because the customer might use Argo already we also have a problem because we can't grant ourselves arbitrary permissions because some of our customers get really upset if you hand over the keys to the kingdom if you will and give everyone permissions and we can't go around proxies either right so we have a few ways to communicate with those clusters and otherwise they might be really isolated so in the end one does not simply walk into Mordor if you don't own the place right but we that's essentially why we couldn't use available open source tooling and instead set off the venture of well building our own thing right and Alessandro will now show how we destructured the problem into smaller chunks thank you thank you Nico again so if you carefully read the title of this talk we spoiled the name of the thing we built for this and it's called package operator so now let's try and see what's like a very high level overview of what's inside package operator and how we could try to help us solving the problems that we highlighted before so as I said back before like deployments are nice because they have some characteristics that help us dealing with pods in this case but what if we could have deployments for whatever we want to get to the cluster right so I think I don't have focus on the window anymore sorry you stole my focus yeah so what we came up with in package operator is a concept similar to replica sets so replica sets handles pods right what we would like to have is a replica set for whatever and we came up with this object set resource that is part of package operator again and which is able to reconcile a bunch of arbitrary objects and aggregate the status via probes so every object set you can watch the status and see if those objects are working as expected or not it's very important that it's immutable and can be scaled to 0 to archive for rollback so this solves already some of the problems tries to solve some of the problems that we saw before and one of the important thing I want to point out is this phase reconciliation thing because let's see it in detail right so in every object set we can define different phases and assigns the resources belonging to the bundle one of this phase in a way that each phase starts only if the previous ones have successfully completed how do we know if they are successfully completed we define probes for each phase that checks if the phase is actually completed right in this case we have two CRDs that needs to be deployed if some of you is not familiar with Kubernetes CRDs they are basically a way to expand the Kubernetes API defining extra entities that allows you to do whatever you want in a way very simple parallel parallelism as you create new entities on a database so you can define your own objects to work with and extend the Kubernetes API so here we have two CRDs we have probes that checks that they are established so those gets deployed and if the probe succeeds for both of them the phase is considered to be completed and only at that point we go to the next phase that contains a deployment that relies on those CRDs right because if we put together in the cluster at the same time the deployment may try to become available before the CRDs are established and something may fail in there because the new entities are not yet established in the Kubernetes cluster so after the first phase is done then the second one the deployment gets deployed the probes check if it worked and then the second phase is marked as successful so replica sets are cool as replica sets are but deployments are more helpful for us because they manage some of the replica sets internals in an easy way so we created the same concept for object set as well and we called it object deployment object deployment coordinates the transition between object sets and keeps the history so that you can roll back like limited history but you can do rollbacks if it doesn't work and creates a new object set when updated and keeps the old one alive until the new one is successfully executed using the probes that you define so this is what is actually helpful and solves most of the problems we saw earlier right then finally we introduce another concept that I spoiled earlier called packages which helps people to create those object deployments and packages are as I wrote in the slide like a single artifact that contains all the manifests, configuration and metadata needed to run an application and as like RPMs or that packages you have a build phase which you basically take your YAML definitions of your resources and put them in a folder with whatever structure you want and add a manifest that contains some metadata about the package itself you can also optionally add the readme file and an icon as you can see on top of the list and most importantly also you on top of like plain YAML file you can also use go templating capabilities to create resource templates that can be templated and then you build the package and everything gets packed into a non-runnable container image that you can store in whatever registry you want and when you want to deploy that package you create a custom resource that can be either a package or a cluster package depending on the scope you want to give to those resources you specify the image that you built before in the spec.image field and then package operator will pick that up and create all the object deployments and stuff for you so here is a brief description of the internals of package operator and then back to Niko for demo time thank you all of this might be a little bit abstract but essentially our goal here was that we take all the smarts that are that you find in like Argo and other open source projects and ship them in one single operator into those customer Kubernetes clusters that we manage so we can then instruct the on-cluster component to do those smart things on the cluster without requiring us to have outside systems exfiltrating data or working at that scale because at this point you can abstract a lot of stuff away and offload that to the cluster directly so what hope this works what you're seeing here is I have a local kind cluster is the font big enough okay font works kind is Kubernetes and Docker so it's super nice to get Kubernetes cluster up and running super quickly I have package operator running here and now I'm deploying something so this is the deployment API that Alessandro just talked about and what you see in here is we have one so this is the NGDX example deployment we have one phase the deploy phase so we keep it super simple for the beginning and in here we have two objects a config map and a deployment like the normal Kubernetes deployment for the actual workload and we define how package operator can make sense of those objects with the declarative probe so here we say okay select everything that is a deployment and check is it available and is updated replicas it's equal it's equal to status replicas which gives us not only the sign is the deployment available but also is it updated spoiler there are a few versions here I'm making it extra interesting so check the plan and create it so it's doing stuff it's immediately available because we preloaded all the images so we see there is deployment there is config map and pods super nice what sets this apart from a lot of other solutions is that when we watch because of those probes and everything being connected if something crazy happens on that cluster like somebody we see that status reflected in our generic application deployment and for us at SREs that's already huge because without looking at any monitoring system without looking at specific application telemetry we can already say something specific is wrong when we look at this deployment immediately Kubernetes is very bad itself and it's going back to work right away but if you happen to stumble over this cluster because of an actual incident having a single resource that tells you where stuff is wrong is already super useful so let's now update stuff that's where things get fun right so what I did just now is I'm doing multiple advanced concepts at the same time because this is the same object deployment I only just patched it with different data so we have a new release but we added a new strange readiness condition so here we see a new probe was added and for some reason I thought it's sensible to say for this to succeed an annotation needs to be equal to a key in the config map a very convenient way of essentially saying wait for somebody to do something and if we look at what the deployment is telling us now we see it's still available that's nice we get back to that in a second and it's progressing why is stopping it from progressing to be to at the moment because hey the deployment phase is failing at the moment because the config maps probe is failing because well that annotation does not exist on that object right and this could be any dependency missing from the deployment so if we something else that is funny here I on purpose renamed the deployment to v2 so again making the jump to a previous talk here at the conference about rollouts, progressive rollout this is a cannery deployment stretch right now we have two versions of this application running side by side until the new version passes all its rollout flags so they run side by side so fine like no customer would see right now that v2 is blocked by something and at the moment somebody is setting this probe right probe will say this annotation needs to be equal to this data key safe bots terminating deployment v2 deployment remains and only the v2 config map is still here because now the new revision passed all its probes and now the old one can go away because new stuff run out successfully they may say ok this is maybe a little bit too complicated when upgrading stuff but where this becomes super handy is spoiler but updating it again and you know mistakes happen mistakes happen all the time and if mistakes happen across thousands of cluster stuff gets expensive so what happened here is a typo in the deployment pipeline somewhere and an image was referenced that doesn't actually exist so a cube can't roll it out that's where we see here there are multiple backups this will never work but you see the v2 deployment is still here in operation and if we check status information we see ok the latest revision is unavailable because the deploy phase is failing because the deployment status condition is not available this is super useful for us because imagine again you are in SRE you are being woken up at 3am in the morning because some cluster is failing you can check this and you see and now if you this is where the demo ends update it to update it again notice the ok v4 is running v2 is gone and v3 because v3 never worked in the first place so we can get rid of v2 was working until v4 took over and now everything is fine again and we wrote a small helper that can also give us a rollout history on that cluster so we see ok the first thing we deployed a few minutes ago worked the second version worked eventually after we patched it the third one well known did never work so we marked it in our history as this was never successful and the fourth one worked in the end and this with just a single tool without needing to set up anything else right is not connected to monitoring solutions this is not needing data to get shipped off the cluster and this is what we are already using in production in some limited cases and want to build more tooling on it so that's what we did time for Q&A come again so the question was if the operator is certified by Red Hat so it's not in operator hub or officially supported by us because that's something we're only using internally we have though the project is open source and we're doing our best to keep open source stuff installable and nicely documented so if you want to check it out and if there is enough interest in it I'm sure we will offer it in some capacity package operator not one but one it's also in the Shed talk description anything else? alright let's repeat package-operator.run thank you very much ok hello everyone so thank you for being here it's the last talk of the conference thanks a lot for you for all the hard work that you have been doing and thanks for Dave Koff it has been really amazing I arrived on Saturday so I don't know for you that have been here from Friday you are heroes it's great I would like to present you some things that we have been doing in an open in an open source wide project in a European project that we have been working so we are going to talk about Edge Cloud Continuum we are going to talk about serverless and we are going to talk about how to design fast applications in visual programming so my name is Jan Giorgio this is a presentation that we have done with colleagues from George and from Luis and this is an agenda for initial and general we are going to start from a high level view of the physics project we are going to talk a little bit about details on the design environment how this is done and then we are going to dig a little bit deeper on how the infrastructure services take place we'll see some performance aspects and then we are going to discuss some optimizations we have been doing in terms of placement and scheduling okay so a physics project it's the acronym so it's the real name of the project is Android Space Time Service Continuum in FAS okay FAS as function as a service some of the main challenges that we have tried to target with this European project so we want to abstract the usage of service offerings okay service offerings and clusters across the continuum so when we talk about continuum we talk about cloud and premise clusters and and when we talk about service offerings we are talking about how to design applications in this type of environments okay and now the users are not always experts okay so that's why we are talking about abstractions here so there could be data scientists that have a specific expertise on how to create artificial intelligence applications but they might not know how to actually deploy them and even more how to deploy them in a complex scenario with edge cloud clusters we are talking about one of the challenges is the adaptation of code in serverless computing paradigms okay so I'm sure you have heard of there are open source serverless platforms there are also service offered in the cloud here we have taken a particular open source approach one other challenge is the investigation of space and time okay space in terms of location of execution and time in terms of duration of execution these are things that we try to tackle and again taking into account the whole continuum things are much more complex optimization of resource selection and operation so we do have different levels the global level taking into consideration the whole continuum the different clusters in parallel and the local level when we go and execute directly upon a specific cluster and finally how to reuse new artifacts how to reuse the particular channels and artifacts that we are going to create okay the goals of the project visual programming environment to create workflows okay with patterns that can be used with semantics that can be increased and how we can facilitate the developers how we can enhance the development experience with this type of environments platform level functionalities that will allow us to orchestrate in a cloud interplay and more provider local resource management in order to be able to offer a specific optimization while we execute the services okay for those of you that are in that where Europe provides funding for projects and this horizon age 2020 project it is a research and innovation action with a project budget funded completely for all the partners that have been that are participating to it and we are 14 partners with large companies such as GFT, ATOS HPE, Red Hat Fujitsu and smaller startups like Byte, Rx Technologies and finally universities like Harocopio and UPM from Spain GFT is the project coordinator and the end of the project is actually end of this year okay so we have passed all those phases and we are now in the final let's say implementation phase of the second iteration okay this is a high level technical architecture of the project and actually for those of you that don't know we separate it into different work packages and as you can see we have these are the technical work packages there are some others that are more related with how the project is managed or how the exploitation is done on the technical side we have the user layer the design environment and function DevOps phase it's the higher level of the schema here then we have the global continuum layer where we have all the platform services you can see here that the T represents the different tasks so within the work packages we have specific tasks that will come and tackle particular functionalities or services and so once we have the different services on the platform layer along with we have the semantics the way to optimize the placement across the different clusters data services and the orchestration we reach on the lower level the local level of the infrastructure where we discuss about we do research about the ways to do resource management placement optimization in the level of a particular cluster either edge or cloud we have different pilots different applications large applications with real real conditions the first one is related to smart manufacturing for increased resilience and interplay we are talking about industry for the whole scenarios with edge and cloud trying to be as resilient as possible each health scenarios with artificial intelligence and predictive scenarios where particular data coming from specific patients can be actually can give results and predictions on how they can be treated and finally smart precision agriculture where we take into account digital queen scenarios and how we can take into consideration different characteristics coming from smart farms such as how we can better control and take into account the resources so for example the water the pesticides that are used and how this can be optimally deployed within the smart farm now if we go a little bit deeper into the physics environment we have first of all what we wanted to do with the design environment is to enhance, abstract and then reach the way someone will create its fast application how we can customize the function environment we gave the ability to the user since this is initially something that will be installed on PC to customize it as much as possible through docker files and stuff like that we simplified the creation of complex functions workflows and this is for people that already have used serverless or fast you may have noticed that for example in lambda you will need to go through a big YAML file in order to provide different the different characteristics of the functions that you need to use and in particular we have cases where you may have applications composed by more than 5 functions so things start to be complicated and imagine even more if you need to do some joints so with a visual environment this is something that we tackle here so you can see in the right here that it's much simpler to actually do the programming like this rather than that we exploit reusable patterns these are things that I'm going to discuss also afterwards increase in mantics and also specific we abstract completely the packaging the testing and the deployment imagine for users that do not have these capabilities that are more experts in coding if they need this to be abstracted this is a feature to use here so these are the baseline technologies that we have been using so I don't know if you are aware of it it's an open source programming environment for even driven applications and so we have taken this as is and we have enhanced it with a specific palette of annotations and particular extensions that can actually facilitate the user in execution upon the continuum I'm going to talk a little bit afterwards about specific extensions and can be used in this context as a workflow orchestration but also as with function execution abilities and it can also be used as a choreographer for specific functions so you can either create your workflow and build it as an image which will be then deployed or you can enable the execution of different functions physics functions directly from Nordred and in parallel we use OpenWisk the open source fast platform which allows us to actually upon Kubernetes clusters the images that we will prepare directly through Nordred okay so this is the how is the design of let's say the local and the cloud environment in terms of the preparation of the life cycle of the artifacts okay so initially what you get is you get three containers that are deployed within your local environment and as I said the Nordred comes as it is and then we have specific patterns and annotations that have been enhanced by physics but we do not break the compatibility we just add those within the palette we have the control user interface and the serverless function generator which actually provides us the bridge towards the cloud the cloud environment where we have the backend which enables us to to take the code from the repository upload the flow to a bucket and eventually trigger a dinking stop with the flow which will in the end prepare the artifacts to be deployed by open risk right so yeah this is the user interface that we can get from physics so through that you can have the function creation the build management, the testing, logging etc there is a video if we have time we can check it afterwards but I will go through some of the characteristics here so we have the view of Nordred directly with all the different annotations and patterns that we can use and by default the language where the different patterns are programmed in Nordred is JavaScript but in particular since there is the support of docker images we can actually have different types of languages directly there and so yeah afterwards once you have built your flow within Nordred environment you can come and check the build that has been done directly from the admin panel of the physics application and here you can come and invoke a particular execution which will then send it according cluster to be executed there is also logging information taking place directly within the environment of the physics application and you also have the ability to connect different flows within specific applications so for example if you may want particular parts of an application to be executed on the same cluster this is something that you can do what is interesting with physics is that we managed to enhance the different capabilities of Nordred with specific patterns that are usually used and demanded in the case of complex applications applications related to machine learning or applications related that need to be executed in a parallel mode in a multi cluster environment so these are things that by default were not treated within Nordred but we have enhanced it for reusability manageability and abstracted functionality so as you can see here as parallelization I am going to talk a little bit afterwards about that with a split join scenario things such as context management you may need to have a specific action a specific container that will share the context with another container being in the same being in the same flow so this has been created in advance or things such as how you can be you can create a retry scenario when you want to save data within an object store stuff such as request management or branch join these are things that have been introduced within the physics palette and because are used and demanded in a more complex scenarios ok so in the case of the fourth join parallelization this is quite interesting because as I said we start from a concept the concept of how we can start let's say we can have an array of inputs then we split this function into multiple functions we have the fast invoker that will come and will deploy a number of containers based on this split and in the end we have a function that will join with aggregate all those results and this concept here is implemented by this complex let's say flow which is then created as a single function ok so in the end when you want to use it you just need to drag and drop this particular box and the user just needs to parameterize it in order to have the result we implemented here ok and this is something that is actually quite huge especially in complex cases such as machine leveling or you know where you want to improve the performance of a particular execution we have also different capabilities here such as you create different functions or you may create different threads within the same functions or different processes within the same container depending on where you are executing ok other semantic nodes that are interesting to mention here is things such as affinity anti-affinity these are things that you may use quite often in scenarios where you have sharing of functions within the same nodes or you prefer not to put them in the same nodes locality of functions you may prefer that a particular part of your application is executed at the edge if you have data privacy issues for example or that a particular part needs the cloud if it needs high performance for example the importance of high-medium low for prioritization optimization goal you may have different demands in terms of performance energy function sizing demanding particular resources these are things that are treated the requirements of the architecture and things such as how to link particular data services ok now if we go a little bit into detail on how things function internally in the infrastructure layer we do use Kubernetes ok ok D or OpenShift we use Submariner for the connection of the clusters we use open cluster management for the setup we have evaluated and using MicroShift for lower footprint Kubernetes devices of course for monitoring we use OpenWisk and we have started evaluating also ok native for function as a service but mainly OpenWisk at this time at this moment and then we take into account we use Kubernetes operators and webhooks in order to manage how the components will run on top of Kubernetes ok so yeah this schema on how this are setup let's say in our multi-cluster environment you see different clusters here the hub cluster has the OGM Submariner allows the connection and then we have the semantics coming giving the description of the resources of particular nodes and now if we check how the function registration takes place we start from workflow CRD ok which gets the first demand of what we need to execute and then we get the object directly from a managed cluster we trigger the reconciliation loop in a workflow CRD operator and this will allow us to register the function within OpenWisk ok and eventually store it within ATCD on the local cluster and on the OGM so that we can be able to track it afterwards and now once this is done then in the in terms of function execution we have we have the execution of function which is triggered within OpenWisk if there is already a hot container there a hot pod then we can actually use it or else a new pod is needed and hence we go through the webhook is going to allow us to enable specific annotations for a particular scheduler and this is something that we will talk also afterwards on how specific schedulers that we have implemented that would allow us to make some optimizations going beyond the image layer that is implemented within Kubernetes and into account the layers of containers and so once this is done then we have the particular annotations related to the particular scheduler we have also collocation annotations such as we want to prefer to have collocation between a specific function or not and then the particular pod is created on the particular function is created up on a particular pod deployed on a specific node ok so if we take a little bit some performance aspects we have done some experiments and some that were interesting to show here we have triggered workflows related to open-wisk sequences of functions we have also triggered workflows with full primitives and intra-container parallelization so in this case we have a particular within the local environment of Nodred we have the function the different functions which are triggered directly by Nodred in the local environment and finally we have the full parallelization directly by the open-wisk interface which can deploy different containers directly on the different clusters ok in these scenarios we have measured how what are the delays that we get in those three scenarios and as a matter of fact it's interesting to see that of course if we are in a Nodred environment this delay is minimal so if we want to test something very fast but with limited parallelization only within the same container then this will be quite quite fast to deploy whereas in the case of the other two the open-wisk cases then we are talking about 100 milliseconds of delay to deploy the actual containers and some other interesting measurements here related to the space-time continuum where we have we have things such as the network latency ok we have three different clusters one that we consider as a small machine located in Greece another one and two others that are more like cloud machines the one is on AWS and the other one on Azure and through the experiments we have tested to get answers about what are the configurations that we have done and if they were correct or not so for example in terms of latency we can see here that the execution of the particular functions in here it was related with artificial intelligence cases of our eHealth application we can see here that it was much faster of course since it was the edge case it was executed in Greece and then we have since the call was done from Greece it was the latency was much smaller to reach the Netherlands on the Azure cluster and a big year to reach the Netherlands which is the other cluster other interesting things were the fact that in the waiting time we had a long waiting time for Azure cluster here and this was because we haven't configured well the memory of OpenWisk and if we had 32 gigabytes of RAM available we had only configured 2 gigabytes of RAM to be dealt by OpenWisk and this was the difference we had between AWS and Azure let's talk about some specific extensions that we have done in terms of scheduling so we have been using okay we have been using so two different levels of schedulers one in the global continuum and another one in the local continuum and the local level and so these both were implemented in the two different layers the first one was actually related to how we can have multi objective scheduling taking into account performance and energy whereas the second one is more related on how we can improve and minimize the cold starts taking into account the layers locality not only the image locality okay and this implementation is currently ongoing and we are in contact with the upstream Kubernetes community in order to push these changes in the mainstream in the upstream version so last slide one of the things that is interesting with this type of project is that we have implemented reusable artifacts okay in the marketplace there is a marketplace that people can use and there you can find the catalog of available artifacts, operators, no dread floats, data sets, semantic models and stuff like that and people can actually test directly okay so thank you very much and if you have any questions I would be happy to answer thank you very much


The pigeonhole principle states that if we have n objects that are distributed across m different places And if n is greater than m Then some of those places must receive at least two objects Let's consider an example. Let's say we have five boxes and we have 20 balls to put in those boxes So we start putting the balls in the boxes for example, we put the first ball in the third box and the second ball in the first box and Let's try to make it such that we Have a box all boxes such that they don't have two balls The third ball in the second box And then one in the fifth box currently none of them have more than One ball in them Now in the fourth box has a ball now. We need to place the sixth ball in a box So that means that at least one of the boxes must have more than one ball in it And that's what the pigeonhole principle states Of course, we may keep going and we may end up with this as one example scenario where there's one ball in box two two in five and Other balls in one three and four So in this case there were four boxes with Two or more balls in them as an extension of that Let's say that instead of placing them as we choose that we randomly place the balls in the boxes So with a uniform random selection of the ball of the boxes to place the balls in then It would end up that on average That four balls will be placed in each box On average if we repeat this experiment of enough times then the balls will be evenly distributed amongst those boxes


much. There we go. Recording in progress.  All right, let's get started. Supply chain security. I have reduced that rather long title in front of app code cloud to just cakewalk. Make it sound easier. Make it sound more friendly. Put some numbers in there to make it sound a little bit more security, because everything is supply chain, like everything, seriously. But I'm going to try to go through ways in which you can get started with making sure you're doing, you know, at least the bare minimum with your supply chain. And I will go code cloud essentially. And I'm going to emphasize a certain phrase there at the bottom, which is be deterministic over probabilistic, which is maybe an interesting way of approaching the way we're going to deal with security, you know, across the board. All right. And by the way, that is a real undocked photo of me in a chef outfit. No Photoshop at all. Let's move.   This is me. That is actually the room I'm in right now, just a different angle. I got Chewy over here in his anonymous mask. Hard to tell. He thinks people can't tell who he is, but we know. I'm a developer advocate at Bridge Crew. I'm a DevSecOps enthusiast for what that's worth. The evidence of which I run the DevSecOps London gathering meetup. So if you happen to be in the London area and you're looking to get more DevSecOps because you haven't got enough, check that out on meetup. I'm a big Raspberry Pi geek. I've got a bunch of pies you can't quite see behind me on the shelf that run all sorts of wonderful CICD experiments. And I've worked in a lot of places, but I've landed at an awesome place called Bridge Crew by Prisma Cloud. I do a lot of other things. I've got a Twitch show whose name I'm not going to say, a podcast. I'm a beer taster. And I, you know, if you just want more of me, which by the end of this, you won't, there's lots of ways to find out more about me. I'm very friendly on LinkedIn. So if you want to connect, go for it. I'm very easy to find. Thanks to my name. All right. Let's, pardon me, set the scene. Security. And I'm going to, I'm going to put this in the context of more traditional InfoSec. Just to try and get your heads around, if you're not in security, where, how do they think? What are they doing? You know, why are they, why are they so much trouble? Security has a tendency to look at the output of what we're doing with security and they see noise is good amazingly.  Well, we all hate noise. But the, you know, the opposite of that is dangerous in many cases. They like to see more information coming at them.  And then they try and produce indicators of compromise, or they try and create a signature or detect anomalies or look at runtime. This is this form of advanced monitoring is, it's, it is kind of late in the game. But the opposite of data means we might miss something. So they consider that to be dangerous. So they're constantly in this cognitive overload state and trying to look for ways to solve that. So let's keep that in our mind. The other struggle that that security has is whether they're looking at a primitive solution. You know, if I didn't say advanced solution, I said expensive solution that tends to be the finance. Maybe there's an open source tool, and I'll be talking about a lot of open source today that does some pretty cool things. I mean, I would be, I would be, I would be lying if I didn't say that I work on open source tools, like check off is what I'm going to use throughout. And it's great, but scaling it to thousands is not easy. Now, obviously, you can pay for big solutions that do a lot of things for you. But then you're looking for return on investment. This is just more cognitive load for them. The result of what happens is that we, they have a tendency to let that overload make them more reactive than proactive. And what I'm going to be siding towards in this talk is how to be more proactive, which is good, and then easy ways in which to be proactive. Probability versus certainty. If we're looking for things, and we don't know how to define them, that's probability. If we can provide some sort of determinism, then we can err towards the side of certainty, and we can do less. You're always going to be doing some probability, but as little as possible. That's what we're going for, right? All right, I'm just going to define risk as likelihood. Risk is likelihood times impact. Sometimes you just need someone to say that. Some might argue it's an oversimplification, but it's a very handy one. Because if you look at everything you're doing from a security perspective, and you can usually categorize it into, am I reducing the likelihood? Or am I reducing the impact of my action? Or am I doing neither? And should I just scrap it? And that's a good thing. That really does help us define what it is we're going to be doing in the way we're taking action. All right, we're going to shift now to the everything as code. Everything as code is great, isn't it? Imagine if we did everything as code. Huzzah! We're done. Are we? Are we done? There's a lot of options for everything as code, isn't there? Standards are a thing. So many standards. CloudFormation, Terraform, Palumi, Cluster, Cluster, API, YAML, CTK, JSON of the originals, JSON and YAML. Yeah, the foundation. Helm, Ansible, Python. I didn't do Python in there. Bash, scripting. There's no end. It just goes on and on, chef. There we go. There's no end. There's so many ways in which we can do something as code. How do I pick? What am I choosing? It's good. There's no wrong answer. Everything as code is a good thing. It's reproducible. It's a wonderful place to be. However, we can make some smart moves when we're talking about everything as code that's then makes security easy. Sometimes we just, you know, ops, people can just get run away with everything as code and choose something that they like. But actually, as security people, if we can influence and say, well, why don't you use that one instead of that one? We might be giving ourselves a real leg up in terms of security. Here's the quote. Security needs a codified way of describing what we are about to do as opposed to monitoring what we did. It's a nice quote, isn't it? You know who said that? Albert Einstein. No, I'm just kidding. It was me just now. I just made it look like a quote, so you'd pay attention. But I think it's a good statement. I think it, and it's something that we're going to kind of, this may be thematic throughout this presentation. Okay, why as code? Why do we want everything to be code? Well, click ops. I don't know if you've ever heard that before. It's kind of a bit of a, you know, pastiche, but making fun of the tacking ops onto everything. Because if you put ops on the end, then that's be a good thing. Going into a console, gcloud, AWS, and just clicking and provisioning and click, for one, the interface changes often enough. It's super complicated. And I'll be darned if I can remember what I did last time. So it's bad. The idea of doing anything imperatively, which is the opposite of declaratively, is difficult. It's difficult to do twice. Even if you make a recipe, you do a bash script and you use an API, understanding all the different API APIs, even if you look at AWS, there's just, it's not consistent. If you've got any experience with that, it's just too easy to make mistakes. You have non-repeatable results. You may not have change control.  And other reasons, like humans, you know, humans, you and me, us, we're not the smartest people in the world. Well, we are the only people in the world, but we don't, we always make smart choices. So let's try to automate away our stupidity if we can do that. All right, let's try an inverter thinking, and this is where we're talking about the heading towards determinism. Let's reduce likelihood. Employee checks earlier and often when people talk about shift left, it's a bit of a cliche for security people. For some people, it's new. Let's scan our anything as code for problems or misconfigurations. If it's as code, well, maybe I can determine what's wrong. So I'm not doing that reactionary monitoring thing, right? That would be good. I can use the concept of infrastructure as code and I can employ its context when I'm looking at things like vulnerabilities. Is this vulnerability in a vulnerable place? Is it deployed in a vulnerable way? Can I get a visual of what this looks like so I can see if I've deployed something and that can change our definition of risk. We're talking about likelihood versus impact. The context is king, reducing impact, speaking of which, leveraging chaos engineering. If your Netflix is very famous for kind of being founders of the, with chaos monkey and destroying things, the idea of chaos engineering in our world, although it's existed for years, like in other industries, we're just as usual copying what other people do. The idea of that's what that cow is not happy with this analogy. Cattle versus pets. This is important. We should consider really in a cloud native world that everything is ephemeral. That means it's nothing persists from the tiniest container to the biggest cloud. I should be able to see an anomaly in a worst case scenario. Things are being weird. I have my entire state from code to cloud in git and I can just say deploy and everything gets put back to the way it was. That is amazing. Now, it's very hard to do easy for me to say, right? Of course, start small, but work your way up towards that kind of world so that if I didn't destroy things, then I can make them come back. Kubernetes itself is designed around being able to destroy a container and have it, you know, have replicas, have services that feed different replicas, load balancing, all of that stuff's built in. So this is a wonderful thing for us for security, but we need to be able to dictate state. All right. This is lesson one, part one, part one in the importance of supply chain context. I told you it was a beer enthusiast. This beer right here, this beer right here, is an imperial step.  You're like, I don't know, do we have imperial Budweiser? You would probably like it if it existed. Imperial means it's strong. That's a 10% stout there, a bit of a bit of a meal. And you can see in the description that I took off their website. I changed the percentage to CVSS scores, the common vulnerability score there. And that's high. That's the highest you can get. 10. This looks dangerous. I would not, I would not, I would hesitate to try one of these or I would do so with caution. I'd be sitting down. Right. So we need to get context in its single form. You know, maybe at a relaxing evening, that would be just fine to have one of these. If we check the deployment of this particular can of beer, let's check it out. Eveningdrinks.yaml, this patterned after Kubernetes YAML, if you're not familiar. We can see all the labels. We can see the dinner was salad. I would not have a salad. Maybe a pasta before this. Three replicas. Watching Amazon Prime, other services are available. The surreal show American Gods is weird. And I had three and 45 minutes of this game. And it didn't dry January. Oh my goodness. This is awful. So this is like an example of the worst possible deployment, which amplifies the risk of this container. See what I did there. Waited a while to bring the container pun in there. But that's the idea. This could be harmless or it could be terrible. Containers can scale. Images can scale to create an impossibly possible to remediate attack service. All right. That's the first time I do this. I'm going to do one later on too. Getting started with GitOps. Yeah. GitOps is cool. Infrastructure is code plus GitOps. That is a solid step towards security. And yet it all sounds like ops, right? It's not really on the radar of security. If it's not in your radar, it should be. This clever individual, Victor Silva, immortalized himself on the Weaver website with this GitOps in one slide. And it's really small to read. So don't worry. It's exactly the text I put there. Git is the single source of truth. Git as the single place for change happens. Runtime is now, from code to cloud, is immutable. We're not changing anything. We're never going into the console. I'm not going into Kubernetes. I'm not exacting into a pod. I'm not doing all that. If I need to make changes, I make the changes in Git. And I use something clever, like Terraform or like Argo CD to push that out into my environment so that the changes only happen in one location. Everything is observable. Everything is verifiable. Awesome. Now this springboards us into security very nicely. I'm going to go back into and talk about the term. I've talked a lot about infrastructure as code. And I give you some good examples there. Whole big list, right? And I've talked about deterministic and I've talked about declaratives. Let me give you a very, I think a very simple example of what the difference is between declarative and just say not declarative. Procedural imperative, very similar. Say I need to bake a cake. This one will bake a cake for me. It's written in bash. I'm going to preheat the oven. I got my oven temperature. I haven't gotten any layers yet. So while layers are less than or equal to two, I'm going to mix some flour, eggs, sugar, style equals fluffy. Oh, yeah. Bake time 30 minutes. Repeat. I've got my layers. Pretty awesome. Nothing wrong with that. That used to be everything. The way everything was done. Let's take a look at this declarative example. I could have chosen Terraform here. I could have chosen anything, right? The cloud formation templates doesn't matter. I could have done it like that. Essentially, this is more like being in a restaurant. This is like saying, I would like cake, please. Two layers and I trust, you know, my favorite, my favorite chef, fluffy sponge. I get chef fluffy sponge to make my layers wonderful. I don't want to know how to make a cake and I'll get a cake. Both results are the same. In one instance, I'm relying on somebody else to make sure the state is retained and provision it and make it happen. So I get the cake. In another instance, it's the instructions. It's a recipe, essentially. Now, there's two big differences between these two. One, there's this concept of item potency, which is a strange sounding mouthful that essentially means if I run the script, does it change? Like, do I have the same result? And let me just explain. If I run the bash script twice, I have two cakes. If I do the declarative version, I have one cake because it's defining state. If I already have a two layer cake, it just goes, yeah, you already got a cake. What do you ask me for another cake for? If there's no cake or there's one layer, it makes a second layer. So it's just a definition of retention of state. It's quite nice.  The other one that's different is that what if I get rid of the picture of the cake? What am I making? Just for a readability perspective. The declarative is nicer. I know I'm making a cake. Over there, I don't know if I'm making muffins. I don't know what I'm doing. So there is kind of that element of it. I'll do a little name drop. Conk, let me just do that. I know I'm doing it. Kelsey Hightower, this is a quote from 2018. So kind of going way back again. He was praising GitOps. But what I wanted is the second sentence that I actually would like to double down on declarative configuration is the key to dealing with infrastructures code. Yeah, it's a big deal. It sets the stage for new tools. Now, I've already lately touched on checkout. That is a tool for scanning infrastructures code, but specifically declarative infrastructures code. And it made security in that place easier. So that's really quite cool. So let's just keep that in the back of our minds. While we introduce the bracket supply chain, layers of our delicious cake. See, I'm still on the cake, still on the cake. You'd think I was obsessed with cake. But that's a beautiful thing, isn't it? Look at that. So what are my layers here? I'm going to be going through layer zero, which I consider layer zero the foundation. It's the buttery biscuit base of our cake, right? You got to have a cloud. If you're cloud native, there's some cloud. It's going to be the first thing you do. Then we're going to switch back to kind of an app sec perspective. And I'm going to look at the application. You're writing some code. You still need to, you still need to write secure code. You know, that has never changed. That's something we've been good at is writing good code. So I think as we move to microservices, we're even getting better at it. It's later to where it gets complicated. The application's friends or dependencies where things have gotten more complicated lately. That becomes a part of our supply chain. You can see how it all connects. Everything has dependencies. Even the cloud itself has dependencies that can report serious vulnerabilities that we need to be aware of. So our supply chain is so complicated.  We're going to talk a little bit about the image itself, which, you know, we're building images and we're pushing them into our pipeline and we're wrapping them all up in a gift basket deployment and we're sending that into runtime in our pre-prepared cloud. Our application goes into our image. But there's the pipeline itself. That's gotten a lot, a lot of attention in 2021. We used to kind of, kind of ignore it. And now it's like front and center. All of this is supply chain. The pipeline is not supply chain on its own. Some people make that mistake to think of pipeline supply chain.  That's where everybody comes together in a little party and when it creates the application that's going to be put into runtime. No, it's just an enabler. It's just an automation system for our supply chain. Our supply chain is vast. I'm going to be going through each of these, talking about the what, the why and the pros and the cons and doing a few little play things on the way. Okay, securing the base. The very bisque cloud. What are we doing here? Well, ideally we talked about infrastructure as code, right? We're talking about everything as code. So what we're doing is we're using infrastructure as code for provisioning. Exactly. We're not doing clicky ops. We're not doing any of that. We're going to choose an infrastructure as code type. I'm biased toward Terraform because it's kind of cloud independent, right? Cloud formation is pretty cool. It has its advantages. No doubt. I don't want to get into that argument, but Terraform just works on all the clouds, which is good. And it's, it's stateful. And it's declarative. Brill. It doesn't matter as long as you're using something that would be great. Then I'm going to start using infrastructure as code scanners for my pre-flight checks. Not too bad. There are lots available. I've got a logo up there for check off. So when I work on, there's kicks by checkmarks, their tear scan accuracy, which is now tenable, I think. There's a bunch of options out there using none of them as bad using one of them. Great. Using two of them. That's a bit overkill. All right. But I, I think there's a real opportunity to be able to do that. And it's super duper easy, right? If I were to go over here and take a look at, just to give an example of check off, check off is a command line tool. It has GitHub actions. It has an admission controller now for Kubernetes. It handles all sorts of different languages. It's just really super easy to run. I mean, in fact, you can just, it's as easy as doing dash F. I'm actually in the Kubernetes repo right now. I can run base deployment. And that's it. I just said, this is the file I want to scan. I didn't even tell it what type of file it was. I didn't have to. And it comes in and it says, this is, this is what's wrong with this. There's a good question in the Q and A that I'm going to get to later on. So just trust me, Diego, I will get there. It's about plumi versus Terraform. So I'm going to touch on that. CSPM. So once we've done a good scan of what are, to make sure we're not deploying anything that's vulnerable, how do I know that it's still okay, right? There's these tools called CSPM, Cloud Security Posture Management. It's another Gartner acronym. Somebody once, I was recently in a conversation where someone said, every time you say an acronym, the developer dies. Maybe not, maybe not untrue. Cloud Security Posture Management is the ability to monitor your, your current cloud runtime. There's lots of different ways to do it. Bridge Crew's a good example of monitoring runtime and actually synchronizing it back with the infrastructure's code GitOps version to find drift. That's pretty cool. But using something is always really good. There are free versions. You can see I've got a version of that. So you can get into it's free. AWS and most of the clouds provide you something to look for misconfigurations. But if you can tie it back to Git and play GitOps, that's what we're trying to do here. That's what we're doing in terms of securing the base. Why are we doing it? Well, we touched lightly on it, right? ClickOps are bad. Humans are bad. We're creating code. Anything that humans create should have some form of, some form of automated verification. The pros, the pros of doing this infrastructure's code controlled and observed change workflow. Boom. GitOps. That's why we want to do it. We talked about chaos engineering. Yeah. Infrastructure's code of any kind is chaos engineering friendly. If I don't understand what's happening in runtime, I don't know. Just redeploy. Do a terraform, reapply. Reduction of dependence on tribal knowledge. And that's kind of a big one. I saw an entire talk based on just reduction of dependence on tribal knowledge. No surprise it was done by an SRE and huge fan of infrastructure as code and GitOps. No surprise. What are the cons? Well, infrastructure as code is code. And so does anybody write it from scratch? No, I don't. I go get modules from the terraform registry. I go get, I go get helm charts.  And if you look at, actually we did a, we released some research just last year talking about the percentage of vulnerabilities in, and we scanned all the terraform registry and all helm and came back with some shocking results in terms of how insecure some of the defaults. People are doing, people are being awesome and contributing in some of these modules, but it's not necessarily the most secure thing. So even scanning your, your third party modules is really important. Another con and a little bit less so at the moment, but I've seen it happen is template squatting. You see this in terms of images, but not so much in terms of helm, but I think it's merely a matter of time where you squat on a known famous helm chart, for example, and you just tweak the title a little bit, change the trust boundaries, and suddenly you're deploying a vulnerable version of what you thought was a good thing. Right. The other con for infrastructure as code is that we don't tend to update it as often as we do application code. So we can do it once, we can scan it once, we could think it's great. And then just the world changes around us and it ages over time. So something to notice that we need to be attentive and have regular scans of infrastructure as code, even if it's deployed securely. All right. Layer one, securing the code, the actual application source code. This is old school stuff. I think I used to work in SAST and it's just scanning the code pipeline, scanning the tools, scanning in the code pipeline to make sure we're not creating more vulnerabilities. It's looking for the unknown unknowns, not the known vulnerabilities. We're looking for the fact that we're not screwing something else up. Humans are involved. This is a stat. One of my previous lives, I used to work for a company called Synopsys and it was a data SAST tool called Cuberty and we had a lot of research and statistically for every 1000 lines of code you write, you create a bug or an insecurity. This was a pretty accurate number over a very large data set. So think about that. If you're an active coder, that's kind of the regularity. So running things like SAST, although they can be a bit of a pain, they can be notoriously slow is absolutely mission critical. There's a lot of very cool things out there like guardrails is a good example of a SAST free SAST solution that will plug into your getups and run all sorts of wonderful SAST options regardless of the language you're using. So that's pretty cool. I think people should do that. The pros, shifting left and finding things that are unknown early is a huge advantage because those things tend to be found in pen testing and that's crazily expensive.  There are fast tools out there like SEMgrep is a great modern one. I should add that to my slide. Gosec for go blank. These are all really good tools. You can see the cons I've got slow and potentially disruptive. Implementation can be difficult. Technical stacks in most organizations vary madly as we move to cloud native. So it can be difficult to standardize. We have to rely on developers to choose their own tools. Few tools actually are also IDE integrated. There's some great ones. I do a lot of Python, so I use cornflakes in my IDE, which is like a wrap around Flake 8. It's great for Python. ESLint for TypeScript or JavaScript. And then when we get into some of the newer languages like Rust and Golang, we don't get a lot of selection in terms of the tools we want to use. But the good news is that microservices tend to be more secure. When I was there when we were looking at monoliths, lots of problems in spaghetti code. A lot less spaghetti code as we move to cloud native, which is great stuff. All right, securing your dependencies. 80% of software is open source dependencies. That's a rising number. That's why we're doing this. I know I'm going in reverse order right now. 100% of open source vulnerabilities, known vulnerabilities, as opposed to unknown, are known to the bad guys. You know, you're log 4Js. You're Apache struts. These are still out there. And within days of these things being announced, there are bots looking across the internet to find them. The good news is that there's loads of things to do this. In fact, we're really blurring the lines between SCA and image scanning these days. A lot of the tools like Trivi and Chekhov, they do this for you for free. As long as they see a package manager like FilePresent, like for Python or NPM or Maven, these files just list all our dependencies for us. That's amazing. So it's making this problem a lot easier. Now, I'm kind of getting into the pro, so I thought I'd switch slides. Finding known vulnerabilities is great. This is the low hanging fruit. Now, it's, you know, maybe we'll catch all the things you do yourself, but known vulnerabilities, there's no excuse for releasing with known vulnerabilities not really. And there's a lot of open source offerings that can allow you to do that. The cons, it can be difficult to prioritize because if you've got a lot of code, it's very, very difficult to consume all of the vulnerabilities that you might have in a large code base. Very difficult. 15,000 CVEs on average are disclosed per year, 15,000. To think like, you're probably going to have some in your supply chain. Really worth knowing that. The questions you need to ask and to find security tools for us, are the dependencies, are they used? Have I bloated my dependency file or my package manager file with things that I thought I needed but didn't. Thus, creating a whole bunch of sort of false flags in my CVE list is at real risk. Remember the beer, right? Can we use context of deployment to devalue some of the vulnerabilities that might even be 10 out of 10s? It's difficult. This is under cons because this is hard to do. Not a lot of products out there really enable us to make that happen. Another good question. I'm going to leave that. So far I'm leaving the questions to the end because I think they're going to be fine. All right, layer three, securing the image. Very similar to SCA, finding known vulnerabilities in both what we're adding to our image and what's in our base image and the dependencies. It really does overlap with SCA a lot. There is some real gray area, I guess you could say. You can also though check Docker files because there are a lot of best practices you can do in terms of Docker files. For example, if you build an image and you don't say user nobody or user engine X say, it runs as root, which is weird, right? If you exec in, being root in a container image opens a lot of doors, it does. It's bad. Yet it's a default. So scanning an image for best practices and vulnerabilities is awesome. Why do you want to do it? Defaults are dangerously insecure. Our member Docker is still there. Build once, run anywhere. Amazing. Great for ops. Great for developers. Introduces a whole ton of user space, OS dependencies with critical vulnerabilities that completely bypass the security team that has focused their efforts on building secure platforms for us to run our Kubernetes and our containers on. Super secure nodes. Terribly insecure containers. That's a bad thing. And we need new mechanisms to make that happen. Scanning the image in our CI, scanning it in our desktop. This is absolutely critical. One image can scale to thousands. I think I said that earlier. What are the pros of image scanning? It teaches best practices. I want to stop there for a second and show what I mean by that. So I'm still here. I like trivia. Used to work at Aqua. It's a good tool. And it's easy. Installing trivia is brew. It's simple. And if I want to scan an image, I don't even have to have it. I can just say scan image. Let's do this. Let's do Alpine. Alpine is a very popular base image. Hopefully that's all visible. And I'm going to see three. Because what if I don't know the most current super secure version, but I know the major version is three. I'm going to do trivia image Alpine three. And this is probably going to script my broadcast because it's going to try and download a library or something like that. It's running a little. This is the demo. There we go. Great. OS Alpine. Alpine three. Thank you very much. I did not know that. Now I can create a docker file where I can pin Alpine three 15 zero as my dependent base image. And I learned that by using trivia. How strange. And I know it's got no vulnerabilities. What might have happened in another world is I might have gone trivia three dot 14. Let's say let's go back in time. I'm going to guess that there's a 13.1. You're probably going to backfire because it's something I'm scanning. I'm peeking and poking around to find a good version of or maybe I'm checking an old image that I used to use and dependencies. Then things can go wrong. What's amazing about Alpine? It's one of the quickest, quickest in terms of updating vulnerabilities. I haven't got that very far back in time. Only the 13 and I have three critical vulnerabilities. So that's a great example of how I can determine base images I want to use. That's all I'm doing. I'm not even trying to, I'm not even looking for actionable results. And yet I can do that with tributes. There's kind of no reason not to be using tools like that. And it does integrate into things like VSCOPE. So definitely check that out. It's really, really simple. The cons of tools like this, it can become security theater. No doubt. Do I scan images I'm deploying? Do I scan my entire registry? Am I using my entire registry? I can just confuse my entire SCA process when I start adding images to my dependencies that are not in the repo or have I made the image yet? You need to have a real strategy around that. And that can become a problem. Usually you need a little bit of assistance or planning around that. All right. Securing the pipeline. This is a big one. And I probably could spend the rest of our time on it. But I'm going to try and be quick. Software supply chain integrity. This is what it means. We're talking about tools like, and I've got some down there in Toto, ChainGuard, 6-DAR, Cosine. These are things, Connoisseur. I was playing with that recently. This is a super complicated and very, very open to assistance if you want to become a part of CNCF supply chain. And it needs work. And it involves signing dependencies, artifact integrity, workflow integrity, which may not be something that is on your radar. It's very easy to game a GitHub action. There's a lot of malicious things you can do in there. Without showing an example, there is a way if you have a GitHub action that triggers on an issue, that you can actually add code to the name of the issue that gets executed. There's a lot of weird things you can do with that. Also, secrets integrity. There's a whole lot of tools out there that look for secrets in GitHub repo or GitHub itself does it for you that look to make sure you're not accidentally embedding passwords or API keys in there. This is all part of your supply chain. And this is something we haven't paid that much attention to over the years. Why are we doing it? Well, the supply chain or the inner bowels of the supply chain, the CI CD servers, they have the keys to the kingdom. They have access to your Git. They have access to your code. They have access to your secrets. They have access to your keys. They have access to deploy into your runtime. They have access to everything. And I think I've already probably said it. The SolarWinds idea, we saw it. We saw it last year. There was a few really good example of man in the middle of attack, somebody finding a weak link and getting into the supply chain and injecting their own dependencies that have malicious content. Ensure your code is still your code. Now, I gave you just to flip back. There's some great projects down there that you should investigate if you're keen on getting more information on how you can do this. A lot of it is in its infancy. And you can see in some of my cons, some of the solutions are very difficult to deploy at scale, but and few commercial solutions are available. Problem is huge, but solutions are on the way. So keep your finger on the pulse of that. Securing the deployment. I actually looked a little bit at that just now. Best practices for Kubernetes objects. Operational risk combined with security risk. Why? Because defaults are dangerously insecure. Of course, they are. Often we have no security context in our defaults. And these bring essential context to the use of all vulnerabilities. Our image deployment, our vulnerability management, there are deployment manifests or our Terraform that is calling it to play code that could be vulnerable gives us some absolutely amazing insight as to whether we need to react in a world where we are flooded with vulnerabilities and how I'm wondering how to manage them. This is essentially the answer to that. Now, I'm going to give you a little quick example of that. Whereas let's go over here. Oh, this kind of sort. Here we go. Here's an example of a Kubernetes deployment. And I'm looking at it in VS code. Now, I think it's pretty good, right? And I'm looking for, well, what's the one of the best practices around this? And I can see down here. So I've been using an infrastructure as code tool. It's, I mean, it's called checkoff. And I can look at this here and I can see, okay, so I'm missing a few things, not too many. And I can use the, I can take advantage of all of this in IDE information. I would go down and take a look at this particular one again. And this I've made a few changes. This one, I genuinely think this is my prod one. I think it's, I think it's pretty much good to go and I'm ready. And I rolled out and I can see ensure service account tokens are only mounted when necessary. Oh, that's a good point. Service account tokens in Kubernetes are mounted by default. Well, I can fix that pretty easy with a single, a single line, amazingly. I can, I can put it up here. I can put it down the bottom. Let's pop it just here. Get my indenting right. YAML doesn't like bad indenting. I save it. I see my security tool starting in the scan. There's a lot of tools that do this that trigger based on save, both in JetBrains, both in VS code. And we're good to go. If you, probably heard to see, but there's a tick box there. Look at that. I have no vulnerabilities. I'm ready to check that in. And I'm secure. And this is the kind of little nudge towards security that are becoming easier and easier, particularly for Kubernetes objects. It's very easy to make that happen. Super simple. What blows my mind a little bit is that if you Google Nginx deployment, the first link you'll get will be the actual Kubernetes documentation. And it says, here's a good example. And it's functionally will work, but it doesn't even mention security context. You're really going to become fundamentally educated to understand how you can do that. All right. What are the pros? Many open source tools. Doing things like this, the way we wrap up our code and our applications, lots of tools can help us. KubeLenter, check off, KubeScore kicks. Many of them are easy to use. Few do what I keep saying. Open source tools that combine image vulnerabilities with their context. Few and far between. In fact, right now I can't think of one. I know that we're working on it because it's hard to do, but I don't know many that really do it. The other thing is that different deployment IEC languages, they can subvert your findings. Now, I'm using customize in my example on the command line here. If you look at the application, I have a base deployment. And I have in customize what it does is it adds. So you have a base and then you can override like a class structure. So I can say just add these things to me. Like if I were to look at my customize overlays, test deployment, that's not really a valid deployment. It's just got replicas. It's an addition to my base. So scanning that for vulnerabilities would be like, hey, what happens? You need a tool that actually understands some of these formats like Helm that need to be rendered in advance. So thankfully, the tool I'm using does that, but keep that in mind. One other thing I would like to mention is that I want to skip past it. Nondeclarative abstraction. So this is where I'm going to go to Diego's first question in there. And that is Pulumi instead of Terraform. Pulumi is excellent because it uses languages you already know. It uses what is called like Turing complete languages to add all sorts of awesome things to loop through and create infrastructure as code. Now, in the end, intermediary, this is an example of CDKates. It creates because the destination wants the declarative definition. You know, CDKates creates YAML that gets deployed. The problem we have with these languages at the moment from a security perspective is that yes, we can scan the results before they go live. But much like the same problem I used to have with TypeScript, when it used to get translated into console, consumable JavaScript, is that if we found a problem, how do you translate that back to the code you've written? You end up with something that is easier for ops development, but much harder from a security perspective to make it easy to show where the problem over there on the right maps to the code you've on the left. Now, do I think tools like that are amazing? Yes. Do I think we have a competent security solution for them? Not yet. They kind of combine imperative and declarative together. So it's complicated. This is where I was saying security and deployment. Abstractions make it difficult to scan the code and trace misconfigurations back to generated YAML, back to source. I'm going to touch on this question. That's a very good question. This is cool. I'll mention Teregrin. The importance of supply chain context part two. So this is the example I was trying to lead up to. If I have a container that has a CVE with a CVSS score of 9.8, it's in a back-end service, no external connectivity, not running as privileged. And I'm using a runtime tool, let's say like Falco, to record a baseline of its process activity so I know anything weird happens. Do I care about it? Probably not. I mean I do, but I'm probably a bigger fish to fry. I might have a CVSS 7.6 and a front-end service that is behind a load balancer with an exposed port and has a base image that like Ubuntu that has all the bad things that curl, W get, and map, and a center. Do I care about that one? Yeah, I do actually. That stresses me out more than the other one. And I think this is the kind of problem that we need to start tackling from a supply chain per perspective. All right, touching on runtime. Maintaining the status security. We're already doing a great job here. Security people are, this is what we've done for ages. Intrusion detection, endpoint detection, like this is kind of what we, this is where we, a sweet spot for some people. There are new technologies like EBPF. I just mentioned Falco and Tracy that can do very unintrusive monitoring of what is happening within our runtime. And in fact, things like EBPF are being extended into not just monitoring, but network policies, service meshes, all sorts of wonderful things that actually are improving performance and security. So I think that's pretty awesome. Code-based runtime. I would say this is the very last moment. Almost like the CD is admission controllers. And there's a lot of very cool ones out there that apply policy, but think of it admission controller. Personally, I think of it like the bouncer at the door to a club. The bouncer is there to make sure that you're not carrying a baseball bat and that you're not wearing the wrong shoes. They don't care if you have a criminal record. They don't know what you've got. You know, they don't need to know your backstory. So applying hard policy as an admission controller can be very punishing. It's good to have things like if you're going to use OPA, use OPA tools all throughout your pipeline, all the way to your desktop. You're going to use check off, use check off all the way. Consistency of policy and security as policy is absolutely important if you're going to start using those things in runtime to look for problems with running containers, running deployments, changes to running deployments, or even blocking entry and anything like that. So there's a real good, there's a strategy and philosophy there that needs to be addressed. Detecting unknown unknowns is you have to have monitoring. You have to have runtime anomaly detection. Expensive, late, and traces misconfigurations back to code, back to code. Very difficult to find things in runtime and identify where you're fixing and get a preaching get ops. How do you trace problems in runtime back to your get? It's hard. There are some new tools out there. One's called your bio R that does tagging in get that you can then see in your runtime. It's one of the few ways that I've seen you can make that happen and you can have it for free. So that's something you can do there. But that's still a little bit of a gap. All right. I jumped in some questions on a moment, but I'll do some key takeaways first. One is that there's no silver bullet. Defense and depth is totally critical, right? No one level of our cake is flawless. Shifting lift is good. It means more people means less, more hands makes light work, essentially. Shift middle, shift everywhere. Forget shift left. Do it everywhere. Everything is code is important. Get ops at rest in runtime and declarative equals deterministic and less probabilistic. So I've overrun a little bit. Sorry about that. I'm going to put the thank you up there. I'm going to look at some of the questions that I've got. I'm probably going to go and reverse. Where do S-bombs fit into these software factory activities? Well, S-bombs can be output in just by just about every tool that detects vulnerabilities, particularly in terms of SCA, but also in terms of misconfigurations. For example, Chekhov will produce a Cyclone DX S-bomb. And what that's good is where we're talking about exchanging software, a software bill of materials for the people who don't know what an S-bomb is, means you've shown that this is exactly what you're using. These are the versions. There's attestation as to what is in your software. And these are absolutely critical for software exchanges in the modern era. And even the oven I just bought came with an S-bomb, believe it or not. So these things are absolutely critical for, let's say, B2B transactions of software.  There's something that says the same question for TerraGrant. What was the previous question? Infrastructure as code security practices inside software architecture design patterns with archetypes. Reusable for those notions of security. I mean, seek more IAC templates. I'm not sure the question. Lowering IAC code maintenance. New TFC syntax. I'm not sure. I'm not sure. It sounds like good advice, but I'm not sure it's a question, so I might skip the answer. But the TerraGrant question is a good question by Yaron. TerraGrant is a little bit like customized for Terraform. Terraform enthusiasts would argue that TerraGrant is no longer required, but it does muddy the waters in terms of scanning for infrastructure as code misconfigurations because I don't know of a lot of tools that can handle TerraGrant. So this is more where I'm coming back to. TerraGrant is very handy for a lot of people. I think I used to love it, but when I got my security hat on, I realized I was creating a lot of problems, a little bit like the blooming scenario. So that's kind of where I'm coming from on TerraGrant. Some people are tied into it, but if you can get out of it, I think it's a good idea because I kind of feel like it's a fading technology. Sorry, TerraGrant. Lowering infrastructure as code maintenance mostly for version changes on providers or huge language. Do I have advice if you're talking about new Terraform syntax, a strategy for lowering the maintenance? That I can't necessarily provide advice on. I don't know. It's a security question either, but if I could, I would. Diego, you have a lot of questions. Tell you what I'm going to suggest is that Slack.bridgecrew.io. Slack.bridge and Anna, if you're there, if you can push them into the chat. We have a Slack channel where you can find me and you can even find our founders of our company who are absolutely live for Terraform and love conversations about how to, how to, how to create strategy around this.  So you can get in conversation also on the CNCF Slack. If you're on there already, then find me there and let's continue the, let's continue the conversation. Oh, I'll save me. Yes, you can add me on LinkedIn. There's the other question. Okay, great. Well, we might have time for one more question before we wrap up. When will CPEs be mandatory and when? And speedics or Cycling DX reports should be when I, I think I read it, when and how I don't know the answer. I'm getting a philosophical question about software bill of materials mandatory content. I think, I think software bill of materials is like, I wish that would mature faster. I think the best way to make sure you get what you want out of those is to join the communities that are developing them because they're all extremely open source related.  Love it. Thank you so much, Steve. Any closing words? So for supply chain is hard. And I think actually what I just said is an important thing. Open source is the way to begin everything. Getting, getting involved with the CNCF, getting involved with the Linux foundation. That is a fantastic way to start your security journey in an easy way and a low cost. And from there, you'll find a natural path to making some of the, the major solutions happen for you. Plus, as you can see from my presentation, a lot of gaps in supply chain. So all hands on deck. It'd be great to see you join us in this, in this challenge. Love that. Thank you so much, Steve. Thank you for being here and presenting on this topic. We are so excited that you were able to join us. And thank you to all the participants for being here today as well. This recording will be on the Linux foundation YouTube page if you'd like to relive the moments from today. And we hope to join you that you join us at future webinars. Thank you so much, everybody.


This research explored the extent and factors leading to the glass ceiling for Pakistani female doctors in leadership positions in both clinical settings and academia. It found that the glass ceiling was a challenge faced by these women in their career progression. The main reasons behind this phenomenon were institutional challenges, family support issues, personal challenges and societal unacceptance. Malicious intent of seniors, discrimination, stereotyping, lack of mentorship and ethnic background conditioning were also identified as contributing factors. This article was authored by Amina Nadim and Raheela Yasmin.


We will be able to show you that, and you will be able to sit on the operators. And then you just fire and forget, right? Systems like this, they kind of wildly give systems. Especially without how you guys have mountains. The more aircraft, the more ships, but we just had more altitude. Then grab here, grab here. I shoot this, then it means the system is ready to launch. So it's really easy to launch, it's really...


I grew up in a three-parent household. My mom, my father, and my grandmother.  Okay, what's the difference? A woman. Okay.  You know, I'm doing this, I'm doing that. I looked apart, but I don't act apart. Okay. A woman for me is handling your own, but knowing how to cater to a man's needs. Right? And I think a lot of times when you get that aesthetic of, like, I'm a boss bitch, like I'm a diss, I'm a dad. No, baby, like, but you can't cook. Okay. You don't know when to be quiet. You don't know how to allow a man to lead.