buckets it's interesting to figure out how you actually get storage made available to your workloads if you are a cluster administrator it's interesting to figure out how do I actually connect with the storage that I have available within my cluster and if you are a storage vendor of course you're probably wondering how do I jump on this container bandwagon and make sure my solution is available in kubernetes so first I'm going to give a little bit of a background then we're going to jump into an introduction of what CSI is and then we'll talk about how CSI works with kubernetes and finally I'll give a status update and please save your questions till the end alright so first off a little bit about the background of what the problem is that CSI solves so let's start with kubernetes kubernetes has a very powerful volume plug-in interface this is an interface that enables block and file storage to be made available on to kubernetes workloads to pods on demand it allows volumes to be provisioned on demand this is something kind of revolutionary that kubernetes started where instead of having to pre-provision a bunch of storage ahead of time whenever a workload actually requests or needs storage kubernetes automatically goes out and creates that storage for you and then makes it available to that workload wherever that workload ends up on the cluster and if that workload gets killed on one node move to a different node automatically moves it over to that other node pretty powerful stuff but it has limitations one of the biggest limitations of the kubernetes volume subsystem was the fact that these volume plug-ins the extensions to talk out the different types of storage systems were all built in to the core of kubernetes they were actually compiled with kubernetes and shipped with the core kubernetes binaries so what's the problem with that there's a bunch of problems it was very problematic for us as kubernetes developers for three big reasons one was security we had a bunch of third-party code we still have actually at the moment a bunch of third-party code that's running as part of core kubernetes binaries so these volume plugin authors, vendors would write code that would be committed to core kubernetes and run as part of core kubernetes and that could you know it's a security issue it could potentially result in stability issues if there's a bug in one of these volume plugins since it's part of the same process as for example kubernetes it could crash a core kubernetes component and quality was our third concern which is kubernetes as a project does not have the facilities to actually test a lot of these volume plugins so for example fiber channel requires very specific hardware kubernetes has an intent test framework but we don't have the hardware or the facilities to be able to test something like fiber channel so a lot of these volume plugins that we were shipping that are part of the built-in core kubernetes are not very well tested we kind of depend on users to tell us when something is broken and have people who are familiar with those plugins come in and fix it so it was problematic for us as kubernetes developers but it was also painful for storage vendors storage vendors hated to have to align their releases with the kubernetes release kubernetes is a beast for those of you who've contributed to kubernetes you know how challenging it can be to figure out what the release cycle is going to be, when the code freezes are going to be what all the tests are that you have to run there are a lot of hurdles and challenges to get involved with that and if your product is completely unrelated to kubernetes it doesn't really make sense for you to have to commit code to kubernetes to make your product work with it and then of course some storage vendors also did not want to open source their code and by being part of kubernetes they kind of had to so for those reasons we were looking for a better solution an extension mechanism for storage for kubernetes when we started exploring this was around maybe early 2016 what we realized was that we weren't alone facing this problem at the time kubernetes was not the de facto container orchestration system there was a lot of other competitors in this space and when we talked to them a lot of them were facing the same challenge we talked to docker, mazo's, cloud foundry all of them had the same problems that they wanted to be able to connect to storage but didn't really have a good way to do so and storage vendors a lot of them were sitting on the sideline trying to figure out what was going on they didn't want to be involved but not wanting to have to build a volume plugin for say docker and then have to go build another one for kubernetes and another one for mazo's it was unsustainable and they kind of just wanted one direction to go in and so that's where CSI was born out of it was a collaboration between different cluster orchestrators saying hey let's just come up with a single standard to allow us to be able to use users so again the goal basically develop a standard for exposing arbitrary block and file storage systems to containerized workloads and container orchestration systems I say block and file very intentionally the reason is storage can mean a lot of things for those of you who are storage experts it is a huge umbrella that covers a lot of things but the scope of what we care about is block and file storage the reason for that is both block and file have standardized data paths in linux that makes it easy for us to develop control path standardized apis that allow us to make storage available to any workload but not have to worry about injecting any code inside the workload to actually do the reading and writing so what this excludes is for example object stories things like writing to a amazon s3 bucket right that is a proprietary data path that is not a standardized gcs google cloud storage has one implementation amazon s3 has another implementation there's a bunch of different implementations there hasn't really been a standard that's baked into the operating system if you want to use object storage you need to have an SDK that you bake into your application in order to use a very specific type of object storage whereas with block and file it doesn't matter what type of underlying storage you use that your application uses or standardized in operating system so that enables us to basically develop a layer of portability that abstracts you away from storage but that's not to say that we want to minimize or make storage the lowest common denominator quite the opposite we want to be able to expose the full power of the underlying storage system but with a standard way to be able to consume it and I'll talk a little bit about that in a moment so what is CSI CSI is just a specification it is literally 45 pages of proto and comments depending on how large your font size is CSI facilitates programmatic interaction between a container orchestration system and a storage system so the key here is programmatic interaction CSI is not designed for you as a user to issue these APIs directly yourself instead you talk to a container orchestration system be it a processor or something else and that system on behalf of you needs a standard language to talk out to storage and that language is CSI CSI is still evolving we recently yesterday added support for resizing volumes and by yesterday we had a launch of CSI 1.1 yesterday which included this feature CSI is of course an open source project and I'll give you some information at the end of the talk about how you can get involved in helping develop these APIs or the implementation of these APIs so what are the primary use cases that we're aiming to solve with the container storage interface they are number one the ability to create and delete a volume, a piece of storage I want a block or file volume that I can use with my workload being able to make that intention known to a storage system and have the storage system return some handle for that piece of storage that I want this volume to be attached to this given node kubernetes and mesos and all these container orchestration systems manage a large number of machines virtual machines, physical machines and we want some way to be able to tell the storage system hey I have a workload that has been scheduled to node B and this volume A I want you to make it available on this node B so make some way to be able to say attach or detach a volume next step is mount or unmount a volume on a node for a workload so this is like the last mile once a disk is attached to a given node how do you actually take that disk and expose it all the way into the container the workload that's going to be consuming it so this mount or unmount call would be a way to tell the storage system hey on this node please make this disk available at this particular path and then we'll take care of the rest of the container now we're starting to add advanced functionality so the first three things are the basic things to make storage available but we're now going beyond that to enable use cases like snapshots so a snapshot is a point in time copy of your storage and the reason it's useful for application developers is you are running a database for example and you want to do some sort of risky operation that you may or may not regret later so what you do is first take a snapshot of what you have do your risky operation and if something goes wrong you can actually undo that mistake cloning of volume if you want to fork the existing data that you have and use it for another instance of an application CSI now supports that as well and then the use case that we're adding right now is resizing which is hey I started off with the volume that's 50 gigabytes I actually need more storage how do I actually expand that volume to be able to use more storage the interface itself is a PC interface and it's defined as protobuf three protobuf services controller node and identity the reason we segmented the calls in these three ways is that we realized that these calls can basically be executed in two different ways either as a controller or as a node if you think about the way that well let's think about a cluster a cluster is a bunch of different nodes and the storage system has probably two components one is the back end which is able to accept calls to for example provision of volume and then a second is some agent running on each one of those nodes that says how do I actually make that piece of storage available on that node so you're going to have a piece of the storage driver running on each one of the nodes as well as a piece of code running somewhere on your cluster that knows how to talk to the storage back end so controller can basically be any call that can be executed anywhere on the cluster because it talks to the storage back end it's not specific to any given node node are the set of calls which can only be executed on a given node so once we've scheduled a workload to let's say node B we can only there are a set of calls like mount this volume to this specific path that can only be executed on that machine that allows us to basically deploy the same volume plugin in two different ways we segment today interface into these separate services and then finally the third piece is called identity which is a common set of calls just to figure out what this driver is and information about the driver and I'll give you a little bit more detail about all of this in a minute next let's talk about naming naming is hard so initially I talked about attach, detach, mount on mount but the reality is that the names of these calls in CSI are a little bit longer and more complicated and the reason for that is that what we realize when we're developing the interface is that attach and detach are very specific operations that may or may not apply same with mount on out there are specific operations that may or may not apply to every single storage system instead what we want to provide is generic life cycle hooks so the key into specific basically points in the life cycle of making a volume available to a container and then it's up to the storage system what they want to do within those life cycle hooks attaches an option but they can do whatever they like whatever makes sense for their storage system we want this to be generic and reusable for all storage systems this is a list of all the methods that are exposed in CSI that they belong to it's a lot of methods let's focus on three first the capability method capability is a way to be able to advertise for a volume plugin what it actually supports CSI basically allows a lot of different use cases but every single storage system doesn't have to support everything a storage system can be very very minimal and support a very small subset of things and the way that the given storage plugin advertises what it's capable of doing are is this these capabilities calls so there's a capability call for each service the first one to start with is the identity service so for the identity service this is kind of like plugin level what are my capabilities the first capability here is controller service so that's basically saying the entire controller service is optional so as a volume plugin author I don't even have to implement the controller service if I don't want to say volume plugin I don't have to implement it I just don't advertise that capability then there is volume accessibility constraints this is advanced functionality if your storage system has volumes that are available to some subset of nodes but not all nodes within a cluster this mechanism allows you to be able to express that in a way that is generic so that the orchestration system can intelligently schedule workloads so let's say your volume is accessible to let's say your cluster is segmented into zones or racks and your volume is only accessible to a single zone or a single rack this mechanism allows you to express that if you don't have any such limitation for your volume plugin you don't have to advertise that next let's take a look at controller so controller let's imagine that you implemented what are the sub capabilities of controller that you can implement the first and most common is the ability to create and delete a volume you might be wondering why the ability to create and delete a volume is optional shouldn't it be mandatory like why is the controller service optional why is the create delete within the controller service optional and answer for that is that again this interface is for the programmatic interaction of kubernetes or another orchestration system with a storage back end it allows kubernetes to programmatically provision new volumes if that capability exists but if it doesn't an end user may be able to go around kubernetes or the orchestration system and manually provision volumes on the storage back end and create objects on kubernetes manually to represent those objects so both those cases are supported we don't want to limit to only storage systems that are capable of dynamic provisioning and then the other capabilities here are things like the ability to create snapshots the ability to clone a volume some of the things that we've already talked about and then on the node side I'm not going to go into the details of any of these if you're interested talk to me afterwards so here's what a sample response looks like so if I'm a container orchestration system I discover a new csi volume plugin I'm going to issue a get plugin info on it the get plugin info response then contains the name of the volume plugin so that I can expose that up to my users to say hey I have this volume plugin that's available to you and some vendor version potentially exposed another interesting call on the node side is node get info so I talked about topology a little bit the idea that a volume may or may not be equally accessible everywhere in a cluster one of the things that we didn't want to do is make assumptions about what the storage system knows about the container orchestration system we wanted to make a strict abstraction between the cluster and the underlying storage so get node info node get info is a way for us to be able to get information from the storage system about a given node what does this storage system how does the storage system identify a particular node the container orchestration system might call this node b but that's just what the container orchestration system calls it the storage system might call it node foo and we need some way to be able to translate that so the first thing that happens when a new volume plugin shows up on a given node is Kubernetes calls this call node get info and finds out what does the storage system know about the container orchestration here is the accessible topology so we talked about the racks versus zones thing this is a way for the storage system to share a set of labels with the container orchestrator to say hey for this given node it belongs to what I call zone foo or you know rack X and Kubernetes doesn't need to understand the concept of racks or zones or anything like that but it still gives the storage system a way to be able to express these concepts in a generic way. So this is getting information about that node once it has that now it's able to actually provision volumes and so yes so the question is if the if a container moves to a location where it the storage can't be moved how does Kubernetes handle that so there's two things there one is if that happens Kubernetes basically makes that pod keeps that pod in pending state saying hey I can't attach the storage sorry but taking a step back if you implement this functionality correctly you won't get into that case because Kubernetes looks at the volume that your workload is using and uses that information as part of scheduling so when it figures out what node to schedule a particular pod it says what are the constraints on the volume that this pod is using and let me use that to figure out where to place this workload so ideally you never get into that case so create volume is fairly straightforward it's the things that you would expect a container orchestration system saying create some storage I would like you to call it this the name is a suggestion the storage system can call it whatever it wants and in the response it gives you the actual handle the capacity is how much storage I want 5 gigs 100 gigs a terabyte parameters is what I kind of alluded at earlier we want to make an interface that enables our customers to be able to use any kind of storage that they want in a standard way but what we don't want to do is make that interface the bottleneck the lowest common denominator for for storage systems there is a lot of differentiation between different storage systems and a lot of knobs and cool features that they expose that don't necessarily need to be standardized but we also don't want to hinder so this opaque parameters blob here is basically a way for a given storage system to expose any number of opaque attributes for a given volume so if I have some notion of let's say encryption or some notion of some performance for a given volume like the ability to control IOPS or locality for a volume or any number of attributes for a given storage system I as a storage system can basically expose those parameters opaquely through the parameters call and that gets exposed out to Kubernetes in the storage class object we'll look at the storage class object a little bit later but basically it has a way for a Kubernetes cluster administrator to set those parameters and define this is the kind of storage I want to expose out to my application developers the other attributes and parameters here are for passing in secret information I'm going to encrypt this volume here I'm going to use the option key the ability to specify a data source you know I want a provision of volume but instead of it being empty I want to provision a volume using a snapshot as a source a provision of volume that's pre-populated and then the topology requirements that we talked about earlier so this is not just intelligent scheduling which we just talked about but also the ability to intelligently provision meaning if I know that somebody is going to want a workload with some constraints influence where that storage gets provisioned which rack or which zone to actually make that decision even earlier so once I have a volume provision the next step is making that volume available to a given node once the scheduler for the orchestration system runs and figures out where the workload should be placed the orchestration system then issues a controller publish call to the storage system to basically say for this given volume the volume ID that you return to me please place that volume available to this given node and that node ID is the node ID that we discovered through the get node info call so this is not like some random node ID that they're getting from Kubernetes this is something that they told us that they self identified with and then finally the last mile is okay I know that this volume is attached or available on this node let me actually make it available inside the container and the way that that's done for mountable file systems is we provide a directory and tell the storage system to actually make that volume available at that path and then we bind mount that path into the container we pass it to the container runtime and say please expose it at this subdirectory and then anything that is written to that subdirectory from the application is then passed out to the storage system and gets basically persisted beyond the life cycle of that container now I talked about that attach or controller publish step but again that is optional so if you're a storage system for whom you don't have a concept of make this storage available to a given node maybe you're running NFS where that concept doesn't exist you don't have to implement it alright so that's CSI now let's talk about how CSI works with Kubernetes this is what it looks like I know it's pretty scary so let's take a step back Kubernetes does a lot of the heavy lifting for you for making Kubernetes work with CSI for the three audiences that we talked about storage vendors focus on how to develop a CSI driver to expose their storage system to Kubernetes Kubernetes cluster administrators focus on hey I want I have this storage let's say NetApp or whatever and I want to be able to make my Kubernetes cluster users have access to this storage so they take the CSI driver and deploy it on this Kubernetes cluster and now users of this cluster have access to that storage and then finally the application developers have the easiest job which is just consume the storage with very portable abstract APIs through Kubernetes let's take a little bit of a deeper look at it and the take away I want from you for here is basically that it's easy for all three of these audiences it's easy because Kubernetes does the heavy lifting so as a storage vendor the three steps for you are one you write an application implementing the CSI specification this is a GRPC interface you go and implement it as an application you containerize this application to make it easy to deploy and then you pair this application with a set of sidecar containers that we wrote the Kubernetes development team and you basically have a set of yamls that you're going to give to the cluster administrators to deploy this is fairly straightforward the asterisk here is that it can get complicated because you have to look at some of the advanced Kubernetes functionalities things like mount propagations and RBAC rules for creating non namespaced objects and things like that so you probably have the most difficult job out of the three audiences but that's okay because this is the value add that you're giving to your users if you are a storage vendor that does kernel modules that makes people's lives harder but kernel modules work with Kubernetes just make sure that you're deploying them through demon sets so that it's repeatable if a cluster orchestration system spins up and you node make sure that the environment there has the kernel module that you want and the way that you make it consistent is by deploying a demon set that will install the kernel module next is Kubernetes cluster administrators you work with your storage provider your storage provider gives you a set of yamls to deploy on your Kubernetes cluster the assumption we're going to make here is that your storage system is up and running and Kubernetes has access to it if you have problems with that work with your storage vendor to figure out how to solve that second is deploying the CSI driver deploying the CSI driver should be as simple as just kubectl apply or kubectl create the set of yaml that your storage vendor provides to you if that fails you let your storage vendor and finally setting up the storage class this is the way that you expose a given storage system out to your users this is what I alluded to earlier with the opaque parameters so if you look at a kubernetes storage class object you have a provisioner field which basically says this is the driver to use and then you have a set of opaque parameters which are the drivers that are specific to this given storage system so in this case I have a driver that exposes a type that allows me to specify the type of volume as an SD that's going to get provision instead of a spinning disk to allow me to control the speed of this particular storage system and if a storage system has a bunch of different parameters and you want to expose different sets of parameters to your application developers you can actually create multiple storage classes for the same provisioner with a different set of parameters and simply give it a different easy name for your application developers to look at so I could create a storage class called fast storage and slow storage both of these storage classes are backed by the same storage system the same provisioner the same CSI driver but they have different parameters that the cluster administrator specified in one case PDSSD another case spinning disk or something else and then that makes the life of the storage or the application developer easier but overall point here is that deploying a CSI driver on kubernetes is fairly easy as well now as an application developer you have the easiest job all the apis that you work with pvc pod all the deployment apis are portable they're not specific to a given storage system they're not specific to any given implementation of a cluster they're portable across all kubernetes clusters so ideally any pvc object that you could use to request storage you can move to any cluster as long as that cluster administrator has set up their storage classes properly that pvc the creation of that pvc object should result in the creation of a new piece of storage for you automatically and then once that storage is created you create a pod or you create a replica set or a stateful set or something using that pvc and kubernetes knows how to basically wire all of that together and wherever your pod gets scheduled the storage will follow and you write to it using standard linux read write file commands and block operations the objects look like this for system volume claim i define how much storage i want the access modes and then optionally the storage class name so i can specify if my cluster administrator exposed two storage classes fast and slow i can select between them or if i really really want to be lazy i can leave this empty and my storage administrator has the option of marking one of the storage classes as default and whatever the default storage classes is the one that's going to get used to provision or create a new volume for you this is going to result in that create volume call that we saw earlier getting issued by kubernetes against the storage system to provision a new piece of storage and then once that new piece of storage is provisioned it's ready for use within kubernetes and the way that you use it is anywhere you have a pod you define a volume you say i want to use a persistent volume claim and the name of your claim has to match the name of the claim object that you have and if you were to take this pod object and persistent volume claim object i want to re-emphasize that they're portable so if you started on let's say kubernetes running on google and you moved it to amazon this would still work you as an application developer don't have to worry about the underlying implementation of the storage system or the cluster it's portable across all of these environments you could even move it to an on-premise environment as long as it's running kubernetes and has the underlying storage class and csi drivers configured properly this i believe is the power and beauty of kubernetes it's the portability that it allows for application developers and i remember your comments earlier about the focus on application developers but it does make application developers lives easier so finally what is the status of this project the csi specification itself was published as a early draft in may of 2017 we released a 0.1 version in december of 2017 the reason for this was very deliberate that we didn't go to 1.0 because what we realized is that it is impossible for anybody to define an interface and have it be perfect as soon as it's released in order for an interface to actually be practical and working you have to have implementations on both ends in our case driver implementations and container orchestration implementations so what we did is release a alpha or beta implementation of the specification have both container orchestrators and drivers implement that revise make breaking changes so for example between 0.1 and 0.2 there was no backwards compatibility we broke the api we made changes and then get to a point where we stabilized the interface and then finally promote it to 1.0 which happened this past december with 1.0 we're offering a guarantee that the specification, the interface itself is stable and we're not going to make any more breaking changes so if you're a story developer and you go and write to this interface you don't have to worry about shit I gotta rewrite my driver in three months because you know something broke your driver is going to continue to work at least until CSI 2.0 comes out so we just launched 1.1 yesterday and 1.1 added functionality resizing is new functionality but it's backwards compatible meaning if you have a CSI 1.0 driver it'll continue to work with newer versions of Kubernetes you just won't have access to this new functionality with an older driver in order to get access to resizing functionality you'll need to get an update for your driver on the Kubernetes side we kind of mirrored the releases alpha support for CSI was released back in end of 2017 as alpha it got bumped at the beginning of last year to beta we added a bunch of functionality since then including remote ephemeral volumes and we finally promoted it to GA in the Kubernetes 1.13 release at the end of last year and we're continuing to add more functionality in the releases that are coming out at the end of this month and all of that is going to be backwards compatible now that it's GA on the storage vendor side there are already over 35 drivers implemented for CSI all the major cloud providers have drivers for their storage systems and a bunch of on-prem vendors including the big names NetApp and such have drivers as well and then other orchestrators who worked with us on the specification Mesos they have an experimental implementation of CSI and they're moving towards CSI soon Docker and Cloud Foundry work very closely with us and they continue to do so they're part of the CSI specification Prover governance body but they haven't actually implemented it in their products yet my understanding is that they're leaning on Kubernetes since they're running Kubernetes as part of their products now in order to to use CSI and then about a week ago I heard about Hadoop Yarn actually implementing Hadoop is not a container orchestrator but the interface is generic enough that they're able to use it to expose storage to their workloads as well that is all that I have any questions and actually before I take questions if you want to get involved there are two communities that you need to be aware of one is the Kubernetes Storage Special Interest Group we have meetings every two weeks if you go to that link at the bottom there you'll find all the information about how to get involved and then the other meeting is the container storage interface meeting to discuss the specification itself and that meeting this is an outer dig it's not once a week anymore it's about two weeks or four weeks we made it less frequent after 1.0 because we weren't moving the design as quickly so I need to update that slide all right Hi I was wondering do you have some generic drivers for iSCSI, NVMe local storage the question is are there any generic drivers for iSCSI, local storage and NVMe so iSCSI there is a generic driver for there's not an NVMe version of the driver yet they're hosted in Kubernetes-CSI and right now there are alpha releases with iSCSI, Fiber Channel and NFS they're kind of community maintained and it's a little bit difficult to get somebody to take ownership for it recently the SIG has been able to find a few people to take ownership of it if you're interested please join the SIG and help us kind of nurture those drivers what we also did was break the driver and the underlying library for those drivers apart such that you could either use the generic driver or you could use the common underlying mount library for iSCSI or the common underlying mount library for NFS to build your own driver so a lot of storage systems are based on iSCSI or NFS and instead of having to write their own mount logic they could basically import that library and build the rest of the CSI driver on top of that instead of using the generic CSI driver for local storage we have an Intree Kubernetes Volume Plugin, local PV right now we don't have any plans to move that to CSI take a look at that implementation but CSI is designed in a way that you could add another implementation of local persistent volume if you wanted to upstreaming upstreaming the drivers to the kernel so these drivers are independent of the Linux kernel are you talking about the Linux kernel these drivers are completely independent of the Linux kernel so they are a user space processes essentially and they're just I have two questions one is simple one again I get confused why this container storage interface is not part of the CIG API projects Kubernetes CIG API so the question is why is CSI not part of the Kubernetes CIGs the reason is CSI is actually not a Kubernetes sub project it is a standalone project so it is it was built out of a collaboration between Kubernetes, Meizos, Docker and Cloud Foundry and was founded as an independent project between those four organizations right now there is the governance of it is that there is an agreement between those four to eventually incubate into the CNCF so it will become a standalone project of the CNCF okay thank you the other one is hard one you skipped over the architecture of the CSI right is it possible to just quickly explain this left that slide in case anybody asked that question you need a GRPC let me jump into that after everybody else's questions and then I can just go over it with the people that actually care about it you said it is not going to be implemented with object storage type but I saw CepFS and Gluster and other things so are we getting there somewhere a little bit so Cep and Gluster expose file block and object file and block are the APIs that are going to be standardized for Cep and Gluster object are just going to have to use it directly more questions? alright so for the rest of you if anybody is interested in a deeper dive of how the Kubernetes system works with CSI feel free to stake around otherwise feel free to head off and thank you very much for your time alright so the green box here is the CSI driver so I talked about if you're a storage vendor you implement the CSI interface and you containerize it so that green box actually is a container that is the CSI driver the next step is deploying in Kubernetes so Kubernetes has a master and Kubernetes has a number of nodes actually it has potentially multiple masters as well but we'll ignore that detail on the nodes you're going to have a demon set pod running that demon set pod is going to be the CSI driver written by the cluster orchestrator and a sidecar container called the node driver registrar which knows how to interact with Kubernetes and then the creation of this demon set object what happens is that this node driver registrar drops a unix domain socket onto the host file system into a specific directory which is a Kubelet plug-in something and the Kubelet which is the node agent running for Kubernetes and that's what's responsible for starting pods and things like that for unix domain sockets when it discovers that unix domain socket there's a special interface that is between Kubelet and this node driver registrar it's not standard CSI this is generic Kubernetes device plug-in registration protocol that basically negotiates between the driver and Kubelet to register it with Kubernetes to say hey I'm a CSI driver and please register me and make sure that if anybody wants to use me this is how they do it there's a second unix domain socket which is dropped from the actual driver onto the host volume the path for that is passed through this registration mechanism to Kubelet and the Kubelet CSI component takes that path and says okay any future calls that I need to make out to you I know where you live and I will issue those calls through that unix domain socket and then this node driver registrar kind of gets sits out of the way it's not a proxy or anything like that Kubelet will issue calls directly to the CSI driver through that unix domain socket so that's for the node component the second component is the master sorry the controller component the way that you deploy this is a replica set or a stateful set size one that means that there's only one instance of this thing running on the entire cluster again it's your driver the same container that you implemented over here bundled with another site car container actually a set of site car containers you have the external attacher, the external provisioner external snapshotter and the interesting thing here is Kubernetes does not directly reach out to the CSI driver directly the reason for that is because what we wanted to do was often times the master is not a schedulable node right what that means is that it's not a schedulable schedulable node so we didn't want to be able we wanted to be able to allow users to install new storage systems on their system without having access to the master let's put it that way so what that means is a user only has access to deploy workloads on their nodes they don't have access to the master so what they can do is deploy this stateful set onto one of the nodes that they have access to and still have all of this work but because they deploy this controller on one of the nodes it becomes a challenge for Kubernetes to reach out and figure out where the controller container is and talk to it one approach would have been to expose this as a service that the master could then discover and then call out to it comes with a bunch of challenges including how do you secure that how do you ensure that that service is only accessible by the Kubernetes master and not by anybody else so instead of introducing all of these discovery and kind of security issues what we decided is to flip the script if you look at all the Kubernetes components and how they work, how kubelet works how the kube controller manager works all of those components they don't have a central Kubernetes entity reaching out to them to tell them what to do instead they monitor the Kubernetes API server and they automatically react and do things so the controller basically has a set of sidecar containers which are essentially Kubernetes controllers they monitor the Kubernetes API server and then they poke the CSI sidecar container to invoke certain functionality so if you look at the external provisioner sidecar container it has a loop that is monitoring the Kubernetes API server and it's looking for persistent volume claim objects that point to a storage class and then the CSI container is the same name as whatever the CSI driver is named when it finds one of those PVCs that's a signal for this external provisioner to say alright I have to provision a new volume so it's going to internally issue a create volume call through a Unix domain socket which is inside this pod to the CSI container to say please create a new volume and then the CSI driver container is going to return a handle the CSI driver is going to return that handle to the sidecar container it's going to be the response to the create volume call and then this external provisioner container is going to take that information and generate a Kubernetes persistent volume object to represent that new volume in Kubernetes and then Kubernetes says ah okay I see that you provisioned a PV object that's pointing back to this PVC that the user requested let me bind them together now you have a PVC repair that is usable within Kubernetes so now you can the user can create a pod that points to that PVC object and Kubernetes says okay I have this pod I need to go and schedule it somewhere okay I've scheduled it to node B I need to make sure that it's attached to node B so what is it going to do? Kubernetes is just going to create another API object and it's going to create an API object called volume attachment on that volume attachment object it says make PVC such and such available to Kubernetes node whatever and then the external attacher sidecar container is monitoring the Kubernetes API it's going to see this volume attachment object it's going to look at the PVC object and it's going to de-reference the persistent volume and say ah that PV belongs to me I know how to handle that attach operation let me go ahead and issue a controller publish call it's this CSI driver through this internal UNIX domain socket and then the CSI driver says okay I know what this volume handle is I gave it to you I know what this node ID is I gave it to you let me make sure I do whatever I need to do to make this volume available on this node so it'll issue some back end call if you're running a cloud provider for example to attach a volume to a given node and then it'll send that response back to the external attacher the external attacher then updates the volume attachment object and then it's going to say I'm done this thing is attached and that becomes a signal for the cubelet the cubelet like every other component in Kubernetes is just watching the API server so it's saying okay I've got a pod that's scheduled to me that pod is referencing a PVC that PVC is of type such and such and I know I need to wait for this volume attachment object that's associated with that PVC to be ready so it's monitoring that volume attachment object as soon as that gets attached that allows the cubelet to say okay I know that this volume is now attached to this node now I just need to make sure that this disk or this volume is available inside this container so it's going to issue a call out directly to the CSI driver that's running locally on that node to say hey this disk this volume ID that you understand please make it available to this given path it could be var lib, cubelet, blah blah blah and then the CSI driver says okay I know what I need to do in order to take a volume and make it available to a given path in my system I will go ahead and do that it does it inside of the container there's mount propagation so that any mounts that happen inside that container are propagated down to the host and so the path that Kubernetes passed in is now actually backed by the volume so then Kubernetes takes that path passes it to the container runtime and says please go ahead and start this container but also make sure that this path is exposed at this path inside the container and then the workload starts and the user's application starts and whatever this path is if the user goes and writes to it it will automatically write out to whatever persistent volume this storage system is backed by and then if the pod dies for some reason either the user deleted it or there was not enough resources on the node for other reasons the pod is terminated, Kubernetes says ah ok I know that this pod should not be running I see that it's currently running let me make sure that I safely unmount the storage for this pod so it's going to issue a call out to the CSI container to say hey I don't need this anymore node unpublished such and such volume the CSI driver says ok let me clean that up and remove those mounts and then the kube controller manager says this volume is attached to the new node so it's going to issue a detach operation the way that it issues a detach operation is it takes that old volume attachment object that it had and it just deletes that the kubernetes external attacher sidecar container notices that deletion and says oh ok I guess you don't need this anymore let me issue a controller unpublished operation against the sidecar container and that controller unpublished then results in a detach of that volume from that given node and then the kubernetes controller manager says ok I need to make this available to the new node that this pod is scheduled to and then it's a rinse and repeat of attach, controller publish, mount and node node publish so that's kind of how it works so CRDs are so the question is can these sidecar containers external attacher external provisioners be implemented as CRDs CRDs are custom resource definitions they're a way to extend the kubernetes API so kubernetes ships with a set of API objects like pod, node and a bunch of other objects CRDs are a way for you to basically specify your own objects and more and more the project internally is relying on CRDs to extend the kubernetes API instead of introducing new built-in types but the API objects alone don't do anything they're just a object something needs to operate on them so in this case the external attacher and provisioners are the controllers and the controllers basically operate on these objects the witch controller? Sure yeah I mean you could define a CRD and a custom controller that does the same functionality absolutely but you don't have to because they're built in so now instead of if you're a storage vendor instead of having to create a new CRD to represent what an attach is what all of these things is and figure out how to make kubernetes emit that and then write a controller to be able to read that and monitor that and then issue the CSI call you don't have to do any of that you just write this interface, the gRPC interface focus on the things that you know which is calling out to your backend and you pair it with these sidecar containers which are basically the controllers that operate on the objects that are already built into kubernetes so the question is how you make sure these containers the sidecar containers and the driver are deployed and how they're running and making sure that they're running so the cool thing is that the CSI driver is deployed just like any other workload on kubernetes so the answer to that is how do you make any workload in kubernetes that's made up of multiple containers ensure that it's deployed ensure that it's running and all of these things in this specific case we use a stateful set which is a way a kubernetes object that lets you say I want to have this a certain number of these pods running all the time so in this case if we create a stateful set of size one that means that kubernetes will start a pod which is made up of multiple containers a pod is a basic unit of scheduling on kubernetes so it ensures that all of these containers will be scheduled together if a given node only has room for one container it's not going to get scheduled we make sure that there's room for all the containers we schedule them together we start them together and if something happens that the pod crashes the containers crash the stateful set kubernetes has controllers that are monitoring the stateful set object it realizes that the container crashed or the pod got killed and it'll create another instance of that pod to make sure that this is always running so the cool thing about all of this is that we're using kubernetes to extend kubernetes these CSI drivers are just like any other workload that you're deploying on kubernetes and this is an interesting point because there's another extension mechanism an older extension mechanism called flex volumes and that was a exact based extension mechanism it implemented mount, unmount, attach, detach but those were implemented as executable binaries and in order to deploy those systems you actually would have to SSH into the node or the master and deploy those binaries or the script onto specific directories within that system so this is problematic for multiple reasons one being just because it's so damn difficult to deploy you actually need SSH access to all of these machines you can't just use kubernetes to deploy it and then of course if you bring up a new node you now need to make sure you have some mechanism to be able to deploy the driver on that node whereas with kubernetes if you deploy it as a demon set as soon as a new node gets added it automatically gets the CSI driver deployed on that node and for some reason there was some repair operation or something that happened that took the node down and brought it back up so with the flex you would have to detect that and make sure you fix your driver whereas with kubernetes you have the demon set pod it will get started, it will do the test to make sure everything is okay and deploy the correct bits in the right places sure and then we'll go back there sure so the question is did you use gRPC for performance reasons or what is the reason behind it it wasn't performance reason it was just convenience it makes it dead simple to be able to implement new interfaces it's relatively stable that was the main driver so what kind of attributes are you thinking about yeah sure so it sounds like the question is given the ability for a storage system to basically advertise how to segment a given cluster let's say zones or racks or whatever is there some mechanism to allow the workloads to basically use that information in scheduling to have some subset of pods go towards a specific wrapper zone and the answer is absolutely yes the mechanism that kubernetes uses to express this topology information is labels just labels on nodes and so what ends up happening is when we get a set of labels from the storage system we apply those as labels to the node so if you want to operate on those for your workloads you simply create selectors on your deployments using those nodes to do whatever you want okay and we are out of time so thank you so much we do have one more session in this track Sonia is going to be coming here in half an hour to talk about container security so come back for that I will not be taking the full hour yeah yeah hello hello hello hello okay we are on we are not on yet hello everyone for our last talk of the day but not the last talk of the track we have three more talks tomorrow we are going to have Sonia Kaptev who is going to talk to us about container security Sonia works for twist lock we have been doing container security stuff as long as Docker has been a thing so they have a lot to say about container and infrastructure security and she will have a lot to say about it couple of things just before we get started, number one is we will be having a Kubernetes birds of a feather in this room immediately after her talk you are all welcome to stay for that we will be talking about asking people to discuss what they are doing with Kubernetes what their problems are how they contribute that sort of thing and the second thing is because we have a lot of die hard people with the track just you are aware of the last talk in this track the 430 talk tomorrow at Sunday we have had to cancel because the speaker got sick so and sadly that one won't be happening we will be announcing that later in the meantime let's learn how to secure our container infrastructure so thank you thank you thanks so much I am really excited to be here so super excited this is going to be a basic talk but just for a show of hands so I can engage the audience a little bit who is everyone is doing something with Kubernetes these days but who is doing something experimental who is full bore into production so for the folks that are in production hopefully you have gotten all of the things that I have mentioned down and they are not a problem so this will be a review for you apologies for that but for those of you who are experimenting thinking about deploying containers cloud-native technologies hopefully this will be a good run-through of things to think about when deploying Kubernetes and containers into production and we will talk about a couple of things so what we will cover today we will do a quick overview of securing a host and what that means we will talk about secure deployment we will talk about the integration of Kubernetes itself and then we will talk about shift left container security anybody familiar with shift left or the concept of shifting left and then we will do a Q&A so the one thing I am not going to cover extensively is Docker security although if you are doing Kubernetes then obviously it is a follow-on but I feel like each one of these topics could be a talk in and of itself and I am hoping there is something covering Docker security as well at the conference maybe not okay well next year maybe I will be back and cover Docker security or I can point you to a couple of resources that we have as well that covers that so let's dive in so we are literally just going to dive right in securing the host hopefully these are just some of the best practices and ways to ensure that what you are doing is deploying Kubernetes in a secure way how many are building their own so you are basically public cloud and spinning up a VM setting up the OS deploying Kubernetes just a couple alright and then how many are using managed so we are more on the managed stuff that is fine the thing to note is that I think all of these still apply and so please don't take for granted that just because you are a managed Kubernetes cloud provider has done all of the good things that they should have done because obviously it is going to very much depend on what it is that you are deploying and what you are doing and how you are doing it so these very much still apply even with managed Kubernetes you are not physically installing Kubernetes itself on your VM so minimal install of the OS there is definitely ways and things that you could put in there that would bloat it and also open up additional attack vectors that you don't want so the first thing is making sure that you are using the very minimal set of services that you absolutely need in order for your app to function and nothing else the next one down is update your packages so the packages that you are using making sure that they are completely up to date and staying up to date on them one of the biggest things that we see in the cloud native security field is that packages get updates or there are new vulnerabilities that are released in the cloud I should say sorry and folks don't update their packages they don't update the version that they are using of those dependencies and that is one of the major attack vectors that we see another big one is development teams will deploy various cloud native technologies and then leave them running and forget about them and you are just begging for somebody to come and infiltrate and then you are going to have to have to be there and who has access to it as well as I mentioned remove any unneeded services so anything that is running that doesn't need to be making sure that that is shut down and not open close any open ports that you don't need another massive one that we see and we will dive into it a little bit more in detail when I cover some of the other best practices but locking down the ports that are used if you just leave it open which is very much the default then you are just asking somebody to start scanning all of your network ports that are open to the wild and start plugging away at your host secure your logins by certificates so having a separate certificate store that then allows login access is ideal and then definitely making sure that you use least privilege and so unnecessary logins aren't available to folks and then intrusion detection systems are great definitely alerting you to when something is noticeable or when someone does attempt to access something on any of your hosts and that will give you some preemptive knowledge hopefully before they are able to access anything within your host so then securing Kubernetes again if you are already doing something with it then these are pretty much review things that should be things that you are doing all the time anyway using RBAC RBAC is great it's great but it's also complex so again back to the whole least privilege concept which is a best practice in any type of development but especially cloud native making sure that you know exactly what roles you have enabled what cluster roles you have enabled what users are then bound to those roles and that no one has access to anything so that's definitely something that you want to sit down and plan out and we'll dive into that a little bit more later when we talk about shifting security left but this is not a trivial task right this isn't just oh well I have my administrators group and I'll plop some of the key folks in there and then I have my developers and they'll have access to the development environment like there's definitely a lot of very intricate design that needs to be thought out in advance especially for name spaces and how those roles and cluster roles relate to your name spaces if you've got a dev stage prod environments throughout then definitely making sure that ideally just service accounts have access to your prod environment and developers have access to just development environments so take RBAC seriously Layer 3 protection is also crucial so Layer 3 is your subnets it also determines how different pieces of your cluster the different pods and nodes all interact with each other and how they communicate together this is another thing I'll show you a little bit more detail on but when we get into some network policy things but Layer 3 protection is absolutely crucial as well because if somebody can infiltrate your subnet and I have a great example of that in a second then they can absolutely dive in on your network and then potentially find anything else that's exposed and so this is kind of like a rolling up type of scenario which is why all of the best practices mentioned on the previous slide really relate because as soon as they infiltrate your subnet then they can do a whole host of other things if you haven't done everything else upstream we'll get into how to configure the master nodes and the worker nodes and some best security practices there I'll also talk about at CD and secrets injection so with RBAC I think we covered a lot of this wireless security model that's additive so there is no deny at all so it's not like you can deny someone from doing things you can only give them access to do things so keeping that in mind as well as you're going through your RBAC strategy is key to making sure that you only add in the things that are necessary and we covered that roles and role binding so obviously you're going to define the role itself such as a reader role or an auditor role and then secure the role binding to particular users and within that role binding you also indicate the namespace that's used and the specific user so making sure that all of that is aligned with what you want it to be and again back to the basic denominator of least privilege across the board same thing with cluster and cluster role bindings this is another super key point no you shouldn't open up cluster access to just anyone it should be absolutely mission critical that they have access to the entire cluster we'll get into some federations stuff as well but this becomes even more so important if you do if you are in a multi-cloud, multi-environment HA type of scenario where you have multiple instances of your cluster running that's just asking for trouble limit access to your nodes we covered this already and then access to containers in the cluster should be able to do that that's just a general best practice that we've seen once you enable access through other methods or APIs and especially in the right scenario then again you're just asking for people to be able to hack into that so network security I touched on this a little bit it's a layer 3 segmentation making sure that only the services that need to talk to each other talk to each other on particular subnets and then employing Kubernetes network policy which is actually super intricate and I want to pop out of here real quick and show you some of the docs if you haven't seen this or for those that are again playing around so this is what that API would look like for the network policies and what you see is you've got the ability to set the namespace the pods and the ingress in the egress which is super important so this limits how any of the pods talk to each other and if this isn't employed I believe it's all just open and we'll get to how to set it to not be open by default but this indicates exactly what IP blocks to can communicate with which pods on which ports again getting back to the basic principle of locking down all the other ports unless absolutely necessary and then validating this with the network policy using only the ports that you need to in order for the containers to communicate between themselves so ingress is traffic coming into and egress is traffic coming out of this is also key because you want to make sure it's expected that certain things that are inputs to particular containers are only allowed in and you don't have files randomly being copied out where you don't want them to be so definitely check this out the network policies is pretty cool real time network profiling so this is something like network scanning or even port scanning there are tons of tools out there that will do this for you and it can effectively show you any kind of vulnerabilities that exist any ports that are randomly open for whatever reason again kind of onion layering right if you have to see them potentially it's even more important to have network profiling of some sort so that you know exactly what's going on with your network and when and then automatic rule creation this can be something that's slip streamed into your CICD process that we'll get into or worked into your pipeline for example where as you add new components to your app there's some sort of automatic rule creation that happens to make sure that you've got your RBAC configured and it's doing exactly what you want it to be doing from a network perspective so CIS benchmarks are folks familiar with these yes I've got one head nod and one hand raised awesome so CIS benchmarks are a very good thing they apply to the Docker engine itself they apply to master nodes and worker nodes and I've got more in here so I'm just gonna pop out again sorry and show you this so there's CIS Kubernetes benchmark and they're effectively if you're not familiar there's scripts that you can run to make sure that you're implementing best practices from a it's an industry just standard basically so there's scripts that you can run in your environment to make sure that you are applying to those benchmarks there's one for Kubernetes it's here on GitHub and then there's one for Docker same thing just scripts they're just best practices you can download them and then just run them in your environment so for the master node configuration any configuration changes to the master node can be done via the apiserver.yml file some basic examples of things that you want to make sure are turned off is anonymous access so it will default to anonymous access if you don't indicate where you're looking for traffic from and so if it can't authenticate the source of the traffic or there's no like multi-factor auth or an auth provider indicated it will default to anonymous if you've got the set to on so turn it off and then insecure bind address this is going back to that network policy example that I showed you in the docs if you don't set the bind address then it will basically be open to insecure bind addresses and you want to make sure that that is disabled otherwise on the ingress you can have folks coming in and potentially doing things to your containers that you don't let them doing the recommendations also to apply the cluster to apply this to the cluster Federation API server as well so what the cluster federation service will do is basically allow you to federate your cluster across multi-cloud environments and so this is great if you're using it for HA high availability or disaster recovery for example or things to lower latency if you've got a global user base which at this point I think everybody does so federation is fantastic just making sure that it's locked down and this again goes back to that cluster role and role binding that we talked about if you're open at the cluster level to a particular thing well then it's not just in one cloud of that cluster instance that you're susceptible to attacks but now that's your entire cluster across all of your multi-cloud environments so this is something to definitely watch out for and lock down at the federation level for the worker nodes some best practices is making sure that the kubelet or the kubernetes agent is configured properly some of the basic examples here and again there are a ton so as you're going through and figuring out what your security policy and approach is going to be depending on what your app needs to function a couple very quick examples of things to make sure that are configured is allowed privilege is set to zero so allowed privilege will basically allow privileged containers to run in your environment so containers that potentially need root this is bad don't do it so set it to zero and this way privileged containers cannot run on your cluster disable seed visor so seed visor gives you a ton of information about your entire kubernetes deployment like what pods you have running what their namespaces are what your RBAC configuration is what your network policy is so it's basically like a map to your entire kubernetes deployment turn it off I mean in development it might be useful in fact it probably is very useful in development as your developers are getting up to speed on exactly how to deploy the app how to configure, how to optimize it but as you move through your deployment pipeline making sure that it's set to off and we'll get I'll cover this a little bit in the shift left as well as it relates to the different stages of your deployment pipelines and some ways to make sure that you're doing you're implementing best practices throughout that process and then finally ensuring proper permissions and ownership to configuration files this is one step further than just the RBAC conversation so the configuration files basically are the heart of exactly what's happening and how it's happening to your cluster and all of your nodes and pods so making sure that only the people that need access or only the services that need access to it for whatever reason have access to it really as you get into production that should be extremely minimal if at all locking down at CD are folks familiar with that CD? yes good okay so obviously it holds a ton of sensitive cluster information so and it's unauthenticated by default that's awesome so making sure that no one it is? do we know when that's changing? tomorrow yeah go home and work on at CD yeah okay the comment was that the unauthenticated status of at CD by default so keep an eye out and know when that's happening because it will likely break you it's also recommended that if you are and I'm assuming everybody is using at CD that you put it behind its own firewall again generally best practice in making sure that all of the good network ingress egress policies are set for at CD as well yeah kind of basic secrets injection don't include secrets in your images this is another massive one that we see from a lot of enterprise companies people just leave secrets there either from the development process or because there's not good hygiene for checking exactly the structure of the images that are deployed into production and so the secrets are left in there they're not encrypted bad things happen use a secret store if at all possible there are plenty of options here you can use the secrets injection at run time or at startup and then as files are in very environment variables that will then dynamically get injected in so you don't have to worry about just leaving them there rotate secrets often this is another good practice we've also had a bunch of customers that have left secrets there for months and then close to years now so make sure that they are rotated and that you have new secrets and then entering that secrets are encrypted at rest is important as well I don't think we've had anyone hacked who had their secrets encrypted at rest so that's good news and then encrypt them from view when a pod is inspected another good reason to turn off the advisor but not just the advisor whenever you're doing any kind of pod inspection and there's plenty of tools that will do that for you to be able to read them if they're not encrypted some basic practices around secrets on that note so who saw this article one, two, okay couple of hands here we go yeah so this goes back to a bunch of topics that I mentioned definitely are back right definitely least privilege encrypting secrets even though this wasn't necessarily secrets but the theory at this point is that they just did a password spray approach and took two to three very very common passwords and were able to try across a large number of usernames and hit bingo so another good thing to note another way that this kind of cements a lot of the information covered is making sure that you have the right gene practices just on basic usernames and passwords because once you get one infiltration or one attack vector that's open this happens so don't do that okay so transitioning a bit from best practices to what else can be done apart from very specific kubernetes security best practices is obviously securing the images and as I mentioned I'm not going to dive in into any of the other best practices pieces but a lot of the same concepts apply and securing the containers as well and then also securing the entire CICD pipeline and so that then brings in the concept of shifting security left and so what that means for those that aren't familiar is moving security practices left so earlier in the software development life cycle with the goal of shifting from reactive to proactive security posture and so what we've seen is that traditionally security is something that's thought of after it's something that the security team gets pulled in once the app is built and ready to be deployed and it's potentially just inspected for some basics and then either shipped or given back to the development team to fix those couple of basics and then deployed and so what this ends up doing is it doesn't allow for the developers to be part of that entire process but also what it does is it leaves the app open to even more attack vectors or attack potential because security doesn't necessarily understand all of the intricacies of the app and what we're also finding with a lot of our customers is that the security teams aren't necessarily as well versed in cloud native as the developers are who are actually building the app in these new emerging technologies and so what's worked really well is and the security folks start that conversation for the development process and making sure that the security folks understand exactly what the app is going to do how the different components of the app need to talk to each other and helping the developers actually build that into the design of the app also a lot of the security tooling that we found doesn't actually surface any of the basic security information in a very dev friendly fashion so it's effectively like the two teams that are speaking in very different languages and very well so what we've seen work very, very well with a lot of our customers is actually getting both of the teams in the room early on another great way that this has functioned is that security is then actually able to work with the development teams to set control gates so basically thresholds that allow the app to move from dev to test to staging to production and then they're also able to monitor in real time and apply any updates or changes that need to happen as well as address any new vulnerabilities that come out and so that way everyone's on the same page as far as what needs to happen to make sure that everything's functioning well in production and nobody's open to any kind of attacks like every lot of this so security by design is really important and it's also important to make sure that all of the things that you have here at front automation's been a crucial key as well whereas much as you can automate these things the better and the less room you leave for human error as much as you can automate your network policy as much as you can automate your RBAC configurations and keep those up to date as folks come in and out of the teams as I mentioned control gates are your friend so making sure that those thresholds are established early on in the process and they're known to everybody on the team that generally helps to make sure that everything's moving along smoothly and you don't have for example some regression fixes that need to happen right when you're ready to deploy and potentially extending your timelines share the tooling don't silo information the more information that both teams can share with each other the better and I think the more understanding it fosters across the board and it makes it easier for everybody to do the job that they're meant to do again getting everybody in on the process early is really helpful and then using common expectations for better production protection so what do we expect to happen if we find something in production what is that process going to look like and not just saying well we did all this good work up front really invincible once we get to production everything's going to be great but really setting up that process in advance to know how it's going to be handled and potentially if there are any intrusions that are detected who gets to do what and how that's root cause analysis and all of those standard processes that would happen anyway and making sure that everyone's on the same page there so containers are great and they definitely empower security teams to shift left successfully than traditional architectures we find that it is easier to get security on board and everybody deployed now into production if we get security involved earlier on in the process and containers inherently are more secure than traditional software development so that's always great but definitely some education is necessary and making sure that the in-house security team is up to date on exactly why that is and how that is and they know how to apply security policies that they've been applying for years now to cloud native I think I would be preaching to the choir here if I covered this but I'm sure you all know the benefits of containers and then security for the entire CI CD pipelines is some things to consider if you are doing this manually or if you're looking at a tool for example to do some of this for you is definitely throughout your build ship run process there's certain things that are going to be crucial to deploying a secure app so things like vulnerability management compliance should be very early on in the discussion, the design of the app compliance is things like PCI or GDPR all of those famous acronyms that we know and love making sure that the app not only follows best practices that we've covered for some basic security things but then when you layer the complexity of compliance on top of that as well some shifts that could happen along the way so making sure that's very early on in the design process vulnerability management is just general knowledge of and mitigation of common vulnerabilities that are out there and making sure that you're staying up to date on those as well there's over 30 close to 40 now industry standard sources for vulnerabilities if you're doing it manually there's definitely a ton of information to comb through and understand and they're getting updated every day it was just two weeks ago now that the big run C bug was found that affects all containers everywhere so things like that are really important to be sure that you're up to date on and then as you're moving into shipping and then obviously when the app is running in production cloud native firewalling so firewalling as it relates to cloud native type technologies taking things into account like the network policy stuff that we covered so firewalling off at cd and things like that that can become quite complicated to manage runtime defense so back to that intrusion detection type of thing and what happens when someone tries to hack into your containers and obviously access control is layered on top of that with our back being one thing but then also authentication as well so just some things to think about from the security perspective so it's a little bit shorter but happy to take any questions question why is it called cloud native firewalling why is it called cloud native firewalling well because it's a little bit it's the same similar concept as basic firewalls that have existed for decades but now it's much more concentrated I would say concentrated right because you're not doing like a vast perimeter firewall that we used to do basically firewall your entire server now you're firewalling individual services for example more questions why not considering that Kubernetes was born in the cloud during the time of security why wasn't security first our back is a thing you add if you'd like it but Kubernetes is meant to be run locally it didn't come local until later it started in the cloud no security Josh do you want to take that one I can take that one first of all let me tell you right now at this point our back in Kubernetes is mandatory so you can't actually not use it you can obviously deploy an our back profile that gives everyone permission on everything but you have to actually do that let's just say that early in the history of container applications security like general infrastructure security was a little bit of an afterthought early versions of Docker had no security anything either yeah I don't know what Docker's doing these days and the other influence for what came to Kubernetes was Borg from Google which was behind several layers of corporate network security so the initial thinking of the people who were developing it was to look at Kubernetes security as a network security problem as people moved into production or towards production with our first versions of Kubernetes I both Google and Red Hat Red Hat had been pushing for some of this stuff for a while because SE Linux so basically you had Google looking at it as a network security problem Red Hat looking at it as yet another opportunity to deploy SE Linux and nobody looking at building the security controls into Kubernetes itself as we got people running into production it immediately became obvious that that was not a good approach and that we needed to include our back we needed to include actual encryption of secrets we needed to include mandatory authentication and a lot of services the you know for that matter I think it was only in 110 that encryption of connections to the API became mandatory before then it was optional and honestly it was optional because in early hackish versions of deploying Kubernetes it was actually very hard to set up right and so we didn't want to make it mandatory if it would be a total roadblock to anyone deploying Kubernetes at all now there's tools to set that up which means we can make it mandatory and it is it's also a fairly common design model to identify what a project is not going to address early on especially in the cloud often times things like security or back end storage are clumped into something somebody else will do I think too just so cloud native came out of the desire to get things out the door faster right and Josh feel free to jump in and correct me where it was developers wanting to get things out as quickly as possible and so it was very much an afterthought it was the how do we get out into the cloud quicker faster spoken like a true infer guy it was piggybacked off of the pretty much the same the same question of why do we have another question I'll be right with you you talked earlier about the difficulties or challenges around cluster role binding and role binding do you have any tips for designing that or what are some concerns you think of when you're starting that project concept number one is lease privilege lease privilege everywhere so just as a best practice just say no say no to everything except what's absolutely crucial to keep things running and we're working our way up from there so if Mary Jane doesn't need access to your cluster don't give her access to your cluster same thing with service accounts unless it's doing something very mission critical it should be a very specific task and it should be defined very explicitly I would say start there any other questions as you said even though we implement this all vulnerability management in the build phase it is still possible that vulnerability or exploits can be found after all the project is done is there any like IDS native to Kubernetes that we can keep monitoring our Kubernetes in containers so we can react to the malicious activities I don't think there's one native as in built in do you mean built in that can or both malicious activity or also the at vulnerability I don't I mean there probably is stuff for malicious activities I don't actually work in the security field myself I don't believe there's one built in there's nothing built in and earlier this week I was at RSA and there was an entire show floor of various cybersecurity solutions but I would say that the market is saturated with IDS solutions is to say the least there's a ton of different options out there depending on what you're looking for the tool to do for you I can't recommend one in particular mainly because I'm biased but I would say definitely take a look and see what it is that you need and what you require and what level of coverage you're expecting things to your point about vulnerabilities that come out post do we critical to keep an eye on new vulnerabilities if nothing else that runs the vulnerability taught us that two weeks ago but in general a good tool will give you insights into when new vulnerabilities do come out it will also be able to tell you which components of your cloud native environment are vulnerable or susceptible to that vulnerability and exactly how to mitigate it so if it's something like hey go update this version of library X to this particular version where that vulnerability doesn't exist if it's something where the vendors already patched it and released the next version that's ideal often times you'll get vulnerabilities that are fresh and so the vendor hasn't even had time although you generally won't get details for about a week to two weeks that's the general grace period for vulnerabilities to have to give the vendor time specifically to release an update to that package ideally not opening up all of the customers that use that package to that vulnerability if everything's public is it 90? depends on who you ask one thing I will mention one of the areas that there has been some focus on vulnerabilities is on the container registry level so both the quay and harbor container registries include image scanning that will scan those registries for known security vulnerabilities usually by matching them against CVEs known package profiles and that sort of thing I mean that's obviously only one piece of a much larger security picture but it's one of the tools another quick just add on to that while Josh is walking another thing that we've seen and as silly as it sounds is we've seen folks download random images off of public registries like Docker Hub make sure you know what's in the image before you deploy it right again it sounds silly but we've had a few customers caught by that where my favorite story to tell is actually there was a legitimate library that somebody was using open source the maintainer couldn't maintain it anymore so we handed it off to a new maintainer that new maintainer had other malicious intent for it and so published a minor update with the malicious code and then published a major update and so everybody went for the major update which now then included the malicious code and here we are so just no and that's why it's just one line item that says you know check your packages make sure you know what's in them but check your packages make sure you know what's in them Hi similar area of question like so can you give us any idea about the runtime defenses like is there any provision for Kubernetes or something baked in that allows you to react as well as require from attack again this is back to the ideas question I don't believe there's anything baked into Kubernetes to do there is oh okay you're saying yes to there's nothing baked in okay there's nothing baked in to do any kind of runtime defense so again that would be a whole host of various tools that'll give you different levels of coverage any of them will give you alerting so they will let you know post attack that hey you've been attacked which is not very useful because now you're attacked there are some that will do preventive or they will do proactive blocking and prevention of those attacks so as soon as they I mean ideally what they'll do is they'll know the exact landscape of your environment and exactly you know based on some of that network policy goodness of the ingresses right and knowing exactly what portion of your app talks to what other portions on which ports what's normal behavior what's not any time that there's not normal behavior it'll be able to prevent and then also let you know hey by the way I just prevented this thing and it looks like it was really bad because somebody was trying to like open a bash on your host so yeah the ultimate solution was yeah to direct and I was wondering if there was something built in okay thank you yeah okay any other questions okay well thank you very much thanks guys for any for anybody here who is using Kubernetes who is looking to use Kubernetes is looking to contribute to Kubernetes we will have a Kubernetes buff in this room starting in about 10 minutes


I live not very far from Google headquarters and I see their cars going around my neighborhood all the time and I was thinking that if Google can drive or way more I suppose can make these cars drive themselves, why are we in networking not doing the same? Why aren't we building networks that completely run themselves? So the idea of a self-driving network is that everything in the network is run by the network as opposed to by humans and I mean that the project I think will be quite a long project from the start of the self-driving car which was the DARPA project in 2004 it's now 2019 15 years we've made great progress we're not there yet and I don't think it takes 15 years but I think we are making progress and the question is you know what's the next step and the next step so that eventually we get there. Well there is this discussion of the use of AI and machine learning in networks there was some tangential discussion on telemetry in networks and I think all of those are really important I think I would rather do things with a purpose so have a use case behind it rather than technology for technology sake but I think those are the technologies really good telemetry about the network really deep understanding what's going on and then the ability to feed that into an engine that will produce the right say decisions and the kind of changes you have to make in the network so that the network runs as desired. So AI, machine learning that could be interesting right now we are using more of a rules based system but eventually we will bring in machine learning and better telemetry is important. So if you think about how this all works ultimately anything that is self-driving whether it's a self-walking robot or a self-driving car or a self-driving network works in a closed loop you get information from the network or from the surroundings you process that information and then you take actions in in this case in the network so we want a standardized way of getting information from the network what is going on you know who's starting to whom what kind of interactions are happening in the network and then you need to process that and then you need to come back and say you need to make this change because we are wanting this kind of results in the network so both getting telemetry and by telemetry I mean everything you know statistics sys logs traps whatever happens in the network that should be standardized making changes in the network has to be standardized and then there's another part that I didn't excuse me there's another part that I didn't touch on which is how do you tell the network what is desired today they call this intent based networking so how do you specify your intent that needs to be standardized as well so now the actual engine itself may vary and different vendors might implement them differently but getting information from the network being able to say this is what is desired so this is the kind of changes I need to make and then making those changes in the network those two parts have to be standardized intent ideally should be standardized and then the engine can be proprietary or implemented in a variety of ways