So, hello, my name is Alexander Bakov, I am working on different things within the identity management area and on free AP project, on Samba and on Kerberos and other areas, SSSD and so on. So, this is going to be a presentation about the general state of the authentication and identity management and Fedora and it can be seen as a kind of joint report from all of these projects. The idea is to cover what we have done in, let's say, last year or two in reality because it's all spread out in time in many areas. So, but the first thing I wanted to start with is the celebration. So, we actually have 40 years of HOSIX API service for identity because the get PWNAM and get GRNAM were added in version 780 and TUNIX in 1979. So, we have 40 years this year of a celebration of probably one of the longest existing stable APIs in the POSIX environment, which is, yeah, which is an interesting thing in itself because despite all the changes on the protocol level, on the storage everywhere, we still use the same semantics behind all these calls. And the name service switch, the mechanism that actually defines that you can plug in different mechanisms was introduced 26 years ago. So, again, we're just off by one with the quarter of century celebration there. On the other side, the authentication APIs like PAM, the API was created in 1995 and standardize it into a single sign-on spec in 1997. So, again, we're dealing with something that happened before the turn of millennia. And looking at that, the NSS and PAM basically represent the last APIs made in 19th century. So, this is a picture of kids finishing high school in Finland where they celebrated by driving in these kind of trucks and showing the weird kind of posters on it and, yeah, representing themselves in whatever kind of costumes they can have. It happens every year and they throw away candies. So, there are people standing around and watching and gathering those candies for about two hours. So, this is in Helsinki, they have a lot of schools and so this procession is going twice and I think it's around 80 trucks or so. So, quite impressive. And if we look into the situation, the network protocols that we use for identity purposes, they all actually from that century. SMB is already 30 years into it, 35 even, right? Kerberos is 30, LDAP is 26 and Active Directory has just turned 20 last year by combining all of these things together. Of course, there are updates to the protocols but the fundamentals are still there and it's amazing what we achieved as the industry with those fundamentals because like 90% of Kerberos and LDAP and SMB infrastructure is probably what runs the whole world with Active Directory. And on the other side is standard API because it's a lot of work among different groups and we have different implementations of the same stuff. So there's NSS, Pamela Dapp that kind of combines all these things together then 12 years ago we started with SSSD and there are rumors that it actually contains the SSSD, it's the naming after the people who were doing it, right? And Steven was one of them.  Notable ones is the Pam windbind for example or RSA secure ID implementations and Kerberos, there are actually two Pamker B5 versions, two separate projects with different code bases but the same naming.  So the yellow ones, orange ones, they come kind of loaded into the Glypsy so effectively they load it into every single application. The green ones they load it through the authentication phase in Pam and this looks like a generally good idea to have this extended but it led to a number of real problems over years because first it's a modular stack, yes, but you have to maintain this modular stack in a certain way and then you have to configure it so of course most people don't like to read the documentation and do configuration by hand and 20 years ago OTH config tool was created to kind of help them with it and it became quickly a pasta of multiple choices that contradict each other and create a funny environment where you don't know what applies to what because there's really no way to tell it, you need to know things behind it. But then the other part of complexity is that effectively most of these modules either on the NSS on Pam site they are configured locally, they cannot consult the network for their own configuration because it's a bootstrap and problem and this means that you have to distribute somehow all the details and have them locally because they load it into every single process they need effectively you have access to the system-wide place where these credentials for example and configuration is stored which creates some security problems and more to that we came to the point where there's a bunch of applications that would like to use the same information store for example LDAP to derive and get the access to information which is ultimately non-POSIX like POSIX API has no call and no way to tell that the user has email account as mean you detail that still something that did not come into need for like 40 years but any application that wants to operate like with the user databases most likely wants to use email or some other identifier that it can use to communicate with the user so this information doesn't come from the system database it means there is a duplication of effort to access it and so on so we have new requirements new technically they are 20 plus years old requirements but they are new in a sense that they don't fit into POSIX API here and with all of this thing we have a standard API but we have variety of issues related to it and one set of issues that I described they are really coming from production side but the other is a long-term issues related to maintenance and development of such things so for example because the way how both APIs work for NSS and PAM there is no execution context isolation other than the process itself so whatever you load in NSS will be accessible to any part of a process they can theoretically pick up your credentials if they can snoop on the on the process content itself which is by default they can because that's their own execution process and then as I said it's a local configuration so scaling configuration across thousands of nodes is a real problem and then because these things exist for quite a long time and open source projects not all of them actually persistent or that persistent in the existence the lack of support because original author decides it's not the topic that interests him anymore and so on the lack of support is a real thing so I said about two PAM KIRB5 modules basically both of them died long time ago were redirected last year one of them and one of them was written because the other died five years before so it's kind of confusing thing that that happens there and then the other part is a lack of so so so fancy technology behind it it's not really attractive to new generations of engineers that are coming out of universities and willing to work with fancy app stores and mobile applications where they might get faster breakthrough into top 1% and so on the dreams are not always coming through but they definitely put us away or far away from from the interest of new developers so probably life is easier with rest and we don't have rest we have to maintain this stuff and we use SSSD in Fedora and actually this year is 10 years universe anniversary of bringing SSSD in Fedora the first build 010 some git commit hash happened in 2009 I was even before I think it was Fedora core 11 it was still Fedora core according to the Koji at that point it's Fedora core 11 yeah well the way it works architecture is still stays basically the same and scalable in terms of how it handles things the most important part is that the actual work is separated from the application that loads NSS SSS or Pam SSS so we don't pollute the executable space of that application we just talk to over the Unix domain socket to the responders somewhere and use cash there's some cheating here because this cash has multiple layers and memory cash is actually mapped into the actual process but that should be very low that should create very low overhead to access it and especially for quickly and often access IDs for groups and users this creates quite fast operation even not requiring the context switches in many cases over years this was extended to handle the offline logins extended to store and enforce pseudo rules as a Linux policies some other rules like in case of free APA there are access controls based on the host and service you access and then multi factor authentication things that came a bit later and smart cards of course so the interesting thing is that we tend to think that Fedora is a bit of on the edge with all of this apparently relate when even more extreme than Fedora in its handling of this architecture so Fedora still has all of these pump care be pump because he has 11 modules they mostly rebuilt by the release engineering when mass rebuilds happen but they are still in Fedora repositories so you could configure a system if you need to use them while well decided to stop doing that and deprecate and eventually remove those modules because of the various factors there's as I say some upstream side effectively that there's nothing to contribute to there are better replacements in some cases and just I think this week on Fedora devil there was request let's let's remove an SCD from Glypsy because well it's obsolete apparently it's not fully obsolete but as I said Fedora quickly losing its extreme frontier and of thinking because people are thinking that time has gone for some of those technologies because there's SSSD SSSD handles all of it yes yes yes yes it's just a different module represents access to it and the other part is that you have to manage it you have to somehow configure systems and that became a mess so if you look at the lower part of this slide it's a screenshot from Fedora bugzilla from Fedora 26 actually where just calling off config in the fingerprint package actually makes your system not accessible anymore so you just uninstall a package and your system disable all logins the reason for that is that the off config dash dash update actually requires you to specify all the original options that you specify there if you want to keep them there is no state and it just regenerates configs based on what you provided there and if you don't know what you provided there's no state you lost it so at some point we decided that we need to do something with it the off config maintainer said I'm done with this call there was need also to port it to Python 3 so there was decision made to redo a new tool looking at the actual use cases and covering those that matter of course you can have customization added but for the majority of people this actually went more or less transparent in Fedora so we had this in Fedora for two and a half years before relate came and I think there were few bugs we fixed them but there were mostly no bugs related that you cannot configure PAM and NSS PAM held up or stuff like that while in in rail we got more conservative customers and they are really wanting to have some easy way to generate configs for them on the off select side so there are two two big profiles the knees profile is is there for historical reasons and mostly deprecated because you actually use it in the high performance computing because of the really small overhead they don't need authentication there they just need fast fast access to these ideas because typically clusters are isolated enough in the trusted environment that they really don't need any authentication around it it used to be this way the net group support is actually using LDAP behind the scenes so there's needs API on the host for the net groups but real implementation doesn't use these protocol anymore so you don't need it and the all select itself is configuring is a set of configuration templates that simply configure the the files that define here but it doesn't do create SSSD dot com for example for you or SMB dot com it's not its task for these you have systems that enroll you into a domain IPA has IPA client install Samba has net ADS join but also it has this realm tool that unifies both of them so realm join the text which the main controller you are using and then uses the appropriate method to that so there is a wrapper called off config that implements kind of the basic options that people use by remapping them to the all select options but not all of them and if you want to add something you just add a new profile in all select and maintain it yourself it's extensible then on the other side for the authentication our colleagues from the security team they work it over years to simplify access to tokens to smart cards to all of these devices whether they are virtual software or hardware that implement pkss 11 protocol the pkss 11 protocol itself API itself is there since 1993 but in 2015 there was new RFC that's kind of unified how you specify access as a string access to device and that made possible to generate a common access to everything by parsing this URI somewhere in the P11 kit layer and then deciding where to go so this was plugged in in all crypto libraries that we have in open SSL through the pkss 11 engine into NSS into new TLS and directly in open SSH because they have their unique needs and again there's another tool new PG which has also its unique needs to access resources so there's now interesting unified pkss 11 stack that you can access on a Fedora machine and if you insert a token like a UB key or nitro key you this token suddenly becomes available at any application at the in the top list and actually on there are other applications that use for example new TLS and open SL and all standard mechanisms work there so instead of following complex kind of configurations complex blogs how to set up things with smart cards we are down to really strings like this so if you have a single device you just specify that you actually want to use something from the pkss 11 token and these are just two separate quotes right so this this is in quotes defines the specific token that you want there but this string they can be logged they are mostly human and readable but the P 11 kit tool and pkss 11 tool they can produce them for you so ideally we should have norm tools that allow you to choose which token to use and in Firefox this is pulling automatically so in Firefox you actually have the UI to choose which token to authenticate with automatically and you don't need to do any configuration it's it's working automatically as long as you have access to the hardware which is the other part that you need to have you dev rules that map write access to token that is into it but on the workstation I think it's it's already there for for most of the things for the logged in user while on the Kerberos site we did two big areas of changes over years one of them is on the client side there was a lot of work to simplify how how credentials are stored how they accessed and how we find out users not users our domain controllers over the network and then after all of this security breaches after the problems with hardware and so on there was a lot of pressure to actually stop using weak crypto hashes stop using the algorithms that really lead it through their life and the other the other part that happened is that yes you can recommend that people should not use them but they still use them because they have software that was installed like in 90s and it works and it is a simple thing but new people who come and implement new applications they don't really follow all of these recommendations always so some of the way to apply this is actually force on defaults to not provide access to to the deprecated crypto and we started doing this with the system crypto policies in fedora two two years ago and now we basically got to the point that we removed deaths and triple deaths from it and RC4 hashes on the life support but for example in the current fedora you cannot use them from the command line with the Kerberos so going over the client side first there is a new addition in SSSD that implements a storage for Kerberos credentials using the so-called credential manager protocol that was originally created by Heimdall Kerberos and reused by Apple in in macOS and MIT Kerberos has support for this for what five years or so but the client support only while the server support for MIT didn't exist so SSSD team implemented the server support of KCM storage so you can now use all of these file directory key ring and KCM storages originally we thought that we will use key ring as a secure method because that's the memory in kernel space that you cannot swap out you cannot access if you're not allowed but apparently there are two problems with it one is that it's a limited kernel memory so if you're a sysad mean that has to access with your credentials thousands of nodes you basically generate thousands of small tickets that populate this area and in the kernel side we get only 40 64 kilobytes space there per element and there is a quota on what you can have there and most importantly for its per user so if you're rude you get it used up by multiple applications easily but really if you run parallel SSH you immediately create thousands of tickets that needs to be stored there somewhere so KCM solves this problem by not storing in kernel but storing on disk and hidden by Unix domain socket access and this solves the other problem the other problem is that the key ring is not namespace it so you cannot use it in containers all containers have access to the same key ring that is on the system if they run with the same ID and and that means that you're not secure with it while with the Unix domain socket access you effectively can you're under UID namespacing and you can apply discretional access quite easily so this works well for the containers access and works well for stuff like desktops silver blue and for example the Fedora toolbox already imports automatically your KCM credential which is default in Fedora if you didn't upgrade from the earlier versions the new installations default to KCM and you automatically get your Kerberos credentials working in the toolbox containers yeah so all of these key rings they they had some use cases behind and they still do and another use case with the dear credentials cache is that in MIT Kerberos credentials cache access treated interestingly from the code that resolves to which realm your ticket belongs to and if you have to credential cache collection and you have just as API based application like Firefox then if it sees a ticket from the realm that you want to access servicing it will use that ticket instead of the one that you want so effectively tries to be closer to the final destination but sometimes it's it's not correct behavior sometimes you want to use the cross realm ticket and an access with your own so that that was breaking some assumptions and splitting things required using different types on the client side as well we had the problem with DNS based finding of the domain controllers especially if you had old Kerberos environment and you had access to those resources without fully qualifying service name so you access it all your systems by their first name let's say SSH server one not server one example dot com this doesn't really work well because client needs to find out the mechanism to redirect and construct the full name it supplies to the server and this to the domain controller and what's worse if you get server one as your name there's ambiguity because there might be multiple server one in different DNS so it works well in a single environment single domain environment but really doesn't work over internet so you need to have some tools to discover the right one and you need to find to have some tools to fall back from the canalization or to canalization and these two things were done in Fedora so one of them is to support the DNS URI RFC and it's used by Fedora project itself so you have this Fedora package if you install it installs the snippet of Kerberos configuration that says hey we have Fedora Kerberos domain controllers there and also use the KDC proxy for them to access securely if you know and the client will use DNS URI to discover them and discover the URI to point to the server the canalization part is now a tree state so this was a big problem with two state on and off when people started using when Fedora project deployed free IPA in the environment where it's all fully qualified domain name but some some environments where Fedora contributors work it used the old like non non fully qualified naming of the hosts and it was failing because the library doesn't know really where you want to go before it goes there and tries it so we introduced a fallback it will be in Kerberos 118 and it is back ported already in Fedora and I think in rail as well and it fixes also use the open shift environment where they typically don't have the right principles in place they have principles issued in the name of a container and not the prince the primary principle how it should be so we have also some work in the KDC discoveries so for KDC proxies because SSSD allows to find out the closest site for you to use but the way how KDC locator plug-in in Kerberos library operates together with providers it doesn't really allow you to say hey I have a KDC proxy here use it it just allows to say hey here is an IP address talk to it using standard Kerberos port which is not not necessarily what you want to have we have with the proxy environment on the crypto side so finally we deprecated desk that code removed in Fedora from Kerberos triple desk as well most of arc for features removed except the ones that some bonnets for the active directory operation its market deprecated and there is a nice addition that in in the Kerberos library basically when you do tracing of Kerberos operations you can see encryption types as numbers and you don't know what they mean so now in Fedora those numbers actually prefix it with the short name of the encryption type so you immediately get and it's not only encryption type but also the pre-authentication operations for example offered by the KDC and the client so you can easily see what is there you don't need to go and search for those obscure numbers the other part that is happening is the SPAC pre-authentication this is one thing that supposed to replace encrypted timestamp pre-authentication which allows completely to avoid attacks or password dictionary attacks it is already in Fedora enabled by default for free IPA installs and if you disable encryption timestamp if you don't have old clients then you cannot be attacked anymore with the password dictionary things the next step with SPAC will be to add their multi-factor support so that you can within the SPAC pass all your token values and authenticate easily that is that is one thing that currently SSSD is complimenting by using the host k-tap creating a wrapper and then using that wrapper to pass your token credentials but it doesn't work from command line if you decay in it you need to do a special mangling there to get it working so it will work then the other part that we have is the authentication indicators it's a nice thing that says okay here's a service it's serious enough to grant access to it only to those who in possession of multi-factor tokens or smart cards and in IPA you can say to configuration that only issue Kerberos tickets to that service if the original ticket that the user owns was obtained with a token with a smart card or with something else so we we allow this differentiation and we currently work together with MIT upstream to do this more flexible there's already an interface in 117 MIT Kerberos that allows us to do KDC policy so that KDC driver like free IPA can make a decision whether to issue ticket or not whether to issue ticket with prolonged time based on what was used for example if you use smart card you might get longer ticket if you use just a password you might get ticket that short time and manipulating the authentication indicators and this will also allow us with SPAIC to add to support for FIDON U2F tokens which is very promising thing because it really isolates you from whoever is handling this but there's still need to write an RFC how this integrates with with Kerberos and the question is not in the code it's more or less time in question so who who has time to write this down yes and the one promising thing I'm looking right now into is to map between different approaches so Active Directory with Azure AD integration with Windows Hello for Business with Windows 10 has support for they call it password less so you don't need a password because they can use your biometry or single issue tokens or TPM in your computer to unlock behind the scenes it really is using smart card authentication of PKE in it so certificate based Kerberos operations but it's not visible to users but the users get a Kerberos ticket that has special asserted membership in the special area of that ticket and we can actually map this information to authentication indicator and therefore allow them accessing resources on our site somehow or we can do the other way around we can provision this information in the MSPAC record so that Windows resources or Samba can see that there are these seeds and they can grant access to so it's it's becoming more transparent moving between different approaches that was client sort of site so on identity services and servers and we have plenty of alternatives we have less in rail than than in Fedora on this front right and there are some news on the directory server for example so it has now integrated plugin for cockpit that allows you to configure and manage instance this is my IPA instance that you can see all monitoring details cache statistics and so on so that we don't need to expose this in IPA itself for example it's you can create records just a normal like LDAP modify LDAP add but from this UI not like users but real LDAP entries that you copy there so it's still hardcore but it allows you to configure some parameters that you otherwise need to create LDFs to to manage them there was a lot of work on auto tuning different caches and so on together with Susie and there's some work also on the usability of 389DS console tools together with Susie and a lot of improvements for performance there will be even more soon there's some cache contentions already track it and fixes are coming so on free IPA side it's basically integration right so it integrates most of those fixes and changes as you can see but there's one notable thing is that you can now with free IPA 4800 in Fedora you can run sample file server on free IPA client as official server and there's a single command utility that configures everything for you so all you need to do is to add your shares into the config file it generates it enrolls machine well you have an IPA enrolled machine then you run this tool it doesn't need any credentials it's enough to have the machine credentials itself so you run it on the route it creates all the needed things registers configures SSSD configures the Samba in smb.conf creates all the needed things and it works in my test it also works for users from trusted active directory domains so you can actually have Windows users coming for shares on the Linux side and there's one thing that apparently works in Fedora is that if you take a Windows machine and try to log in into it with IPA user it will work and it will be able to access the shares from IPA but only if it's not a domain controller on the Windows side because there you have to have some additional things which are not in Fedora yet but we are working on them then there's an interesting thing the hidden an advertised replicas which was a request from Red Hat IT which run free IPA on the scale they want to have systems that they remove from customer use but for example run back up on them so now free IPA has a mechanism to mark a replica as hidden if client is enrolled directly into that replica it will continue using it but if it's doing the discovery it will skip and we have added a lot of changes in the certificate management one interesting one is that we have now is ability to supply a snippet for initial dock tag configuration so you can say that I want to have different key length for the CA certificate yeah I want to have it stored in HSM that doesn't work yet but we are working right now on this so hopefully by next Fedora 32 actually release we might even get that and we also have support for IP addresses in the certificate so that should help guys poor guys who run home routers and want to issue certificates for them without using names DNS names in those certificates or open stack that wants to use it and we have health check utility that verifies I think several hundred different conditions that your server might be under and there's a work on going to have recommendations on it how to fix things yes so it will detect that there are conflicts but on a single master you run on all of them and collects all together but it for example identifies broken DNS in trust and that kind of thing that users are not resolvable and this kind yeah it tries to help to point out where the problem is and finally in Fedora we have Ansible free IPA which is a set of roles and tools that provision client master and replica and we started working also on the real management parts like add-in users groups and all this stuff seems to be quite popular now so people report on it and use it so on Samba site Samba 410 it has full Python 3 support so no Python 2 anymore so we got through it took five years yeah yes json-based logging so you get information about users that were logged in into your domain controller and you can then transform it into some other things and do central logging with it there's a lot of improvements on GPO support on Samba AD site and yeah offline yes we have in two things one of them in SSSD if you enroll into Active Directory directly it supports GPOS the other thing is that if you you deploy Samba AD it uses GPOS there so it can provide for Windows clients GPOS and you can manage them then there are some improvements on the LDAP server in Samba like parallel operations and page control added because Samba has its own embedded LDAP server so work in progress very promising is the unification of crypto replacing the homegrown Samba crypto implementations with no TLS that gives us aside from some compliance with FIPS and and so on attempting to get FIPS we also get performance improvements because of optimized crypto and in some cases we get an SMB3 encrypted stream faster than SMB3 checks zoomed stream which you did not encrypt you just checked zoom and the faster means two three ten times so we get really good performance and the same is being added right now to Sieve's client in in the kernel so between the Linux client and Linux server we get boost of performance and the other the other part is that there is a composition in the Linux client in the sieves.co added the operations are kind of combined in a single request that has multiple operations in it and that reduces number of file operations you need to send it so you you actually get something like 30-40% boost over NFS compared to NFS so once we complete the other part the last part the POSIX extensions for SMB3 protocol we will have an ideal replacement for NFS for home chairs for example high performance access so we are very very close to having all this stuff and I'm done thank you


So far in this video series, we focused on setting up the basics, getting the command line, getting Python installed, getting a text editor ready. Now there's a nice couple of extras that we can add on that'll make your workflow a little more smooth as you get into some projects. First, we're going to talk about a handy tool called PIP. PIP is great for installing packages. In Python, there's some great functionality that comes baked into the basic tools. When you install Python, you have all of that already. But PIP lets you download other packages, things that other people have written that let you extend what Python can do.  If you've ever watched The Matrix, this is like Neo strapping into the chair and downloading Kung Fu. Just like that, you can pull in a whole new capability to your Python. To use PIP, go to the command line and type Python 3-M PIP. Install the name of the package you want to install and a dash capital U. PIP is the important part in the middle here. Python 3-M says Python 3, go to PIP and run it as a module. What this does for you is it makes sure that the PIP that gets run is the one that's connected to your version of Python that gets called when you type Python 3 at the command line. Because there can be multiple versions of Python on a single machine, sometimes just running PIP by itself creates confusion because it can install packages into a different version of Python. And then when you go to use it, it's not there and there's a lot of head scratching and posting and searching on Google and it's very painful. To avoid all of that, whenever you use PIP, type Python 3-M PIP. Install is the command that installs a package. In this case, numpy is the name of the package that we want to install. Dash capital U is an optional thing that you can include that makes sure to install it just for you. If you have a machine that multiple users have access to, it'll install it just for your profile. If you're working on a machine that you have complete control over, you can leave off the dash U and it will ask you for probably a root password to install on your machine. But you can avoid all of that by just using a dash capital U. When you run this, your machine goes, reaches into the internet, grabs a copy of this package, downloads it onto your machine, unzips it, installs it, makes sure if there's anything else that it needs, another package that it needs, it'll also go out, download that and install that too. Those are called dependencies, so it'll get the package, all of its dependencies and install them all. Here, we see a little warning that says, hey, the version of pip you're using isn't the absolute latest, here's the command that you could use to update it. Now, we can go and open up a Python 3 interpreter and import numpy, the package that we just downloaded. It's now there available for us to pull in and use and it gives us all of the superpowers that comes with it. As for what it does, we'll use it in later courses and projects, but for now, this is all you need to know to be able to install new packages. Now, they don't come out of thin air. If you want to see what packages are available and get a little more information about them, you can go to pypy.org, the Python package index. This is a repository on the internet where you can or I can or anyone can write a Python package, upload that code and put it there for anyone else in the world to download. It's one of the beautiful parts about Python being open source, there are these tools around it that let us all write and share code with each other and make Python more than it was originally. Here, we can do a search for numpy. We can click on that and we can learn a little bit more about it. In the description, we see that it's good for working with n dimensional array objects, so arrays of 1, 2, 3, as many dimensions as we want. And there's some good tools for doing things like linear algebra and Fourier transforms and random numbers and a whole lot more. This will come in handy when you're working on a new project. Very often, a project will require one or two new packages that you haven't used before. So you can go to the command line python3-m pip install and then the name of the packages and it'll download them to your machine and then you're ready to go.


Live from the Computer History Museum in Mountain View, California. It's theCUBE, covering DevNet Create 2018. Brought to you by Cisco. Okay, welcome back everyone. This is theCUBE's live coverage here in Mountain View, California for Cisco's DevNet Create. It's their cloud native developer ecosystem. A new initiative, only year and a half old. Great cloud native DevOps oriented. I'm John Furrier, your host with my co-host, our next guest is Damon Edwards, Chief Product Officer of Rundeck. Welcome to theCUBE, good to see you again. Yeah, good to see you again. So, you're just on stage giving a talk. I was. About ops, dev ops. I was bumming people out. That's what I was doing. So a lot of all the Cisco early stuff was about new products and new toys and new awesome stuff. And then my talk was about how silos and tickets ruin everything, right? That we've got all these great advances on the dev side of the house and delivery side of the house and the new technologies we've got and everything's high flying and going to be perfect until it all hits operations and things tend to go wrong. So I walked through a bunch of kind of names or chains to hide the not so innocent. We went through some incidents and just sort of the tales of woe and how the disconnects and the basically the siloed way of working. Number one, kind of we group like with like and operations very siloed. But also number two, that we run our lives through these ticket driven request queues, right? And request queues or queues in general, if you look on the product side and then the sort of the physics, the queuing, the queuing theory behind it, queues are economically very expensive things. You know, they add a lot of delays, they create a lot of bottlenecks. They, everyone's, you're, I asked you to do something and I write it down. You take it off the queue, you know, a week later, the context is all different, right? So we have all kinds of bottlenecks, all kinds of quality problems, all kinds of delays. And it's just an expensive way to work, yet that has become the de facto way that we run our lives. And instead of using tickets for what they're good at, which is handling problems, we use tickets as the general work permission system for the entire operations organization. And it's, you know, no surprise that silos and ticket driven request queues that we get what we get. And so what we talked was about how to say, well, how can we get, stop using tickets as the primary way of doing things. How do we read, how do we kind of look at the organization and remove the need for handoffs, right, between those silos, and then replace where we can't get rid of the handoffs with self-service, right? Pole-based self-service interfaces where people can get what they need to get done, do those operational tasks for themselves, and then move on, right? That's what it's all about. So tell us a little bit about what your company does and how you're solving this problem. Cause it's definitely a problem that's out there right now. And people aren't talking about it a whole lot cause it's kind of the ugly underbelly of development knobs, you know, they're trying to solve it, but they don't really want to talk about it. It's less sexy because you get a promotion for delivering the next big project, right? Saying you fixed kind of how operations work, it generally doesn't, you know, the board of directors doesn't know your name. So that's kind of problem number one. But how Rundek factors into this is we make tools for SREs and systems administrators to number one, organize all their scripts and tools, connect all the scripts and tools that the platforms they currently have across those silos, create standard operating procedures. And then probably most importantly, use the access control features to start to give access to people who are traditionally outside of those operational boundaries. Let developers participate in operations. Let QA participate in operations. Let business analysts participate in operations. All those requests they normally have of operations create those services, let them, you know, let them do them. By doing that, you're creating more capacity in operations to focus on issues that you really need to be solved. And you're making everybody else happy because you're staying out of their way. They can move faster, have fast feedback, higher quality, all that stuff. You know, we've done a lot of crowd chats and we've had the questions come up. Is it the culture or is it the tooling? Or is it the people? Look at all the above culture. Everyone goes to the, that's low hander. Yeah, the culture got to be there.  Can you talk about some of the things that you've seen that works? How does someone go, hey, first it's self-awareness. We got to change this. And so if someone's into that mindset, I want to move to the new model to be more agile. To actually streamline those silos and that ticket system. What tooling do they need to use? What are you guys providing? Where's the steps? What's the sequence of tooling and adoption and picks and shovels? Number one, use what you have, right? So this idea that, oh, okay, we're going to solve this problem. We're going to teach everybody to use this one tool. Because everyone's got to learn this DSL or this kind of language. This never works.  We all know the name. A couple years ago was another tool. We all know the name. You know, these configuration management tools. Now we're on to the new container world. They're like, I don't know if we need that or not. Everyone wants to do what they need to do. So let them do what they need to do. It's a very lean idea. Focus on how to connect those things. Focus on how to orchestrate and organize what you've got already. And then from there, focus on how do we do two things? Limit those handoffs. So that kind of is more of an organizational issue. And number two, all those handoff points. Anything I need something from you, John.  I don't want to have to say, do this for me. You do this for me. And I'm going to wait. And you've got five other things you're working on. You should create services that I can pull from. I need something from you. I need something you would normally do. Hit an automated service.  Do it the Amazon way, right? Which is, I can hit an API and get what I want when I want it. And most importantly, it's not just a one way button I can push, but I can actually create those buttons myself, right? So I can give the thing that I need to do. You can look at it and say, yeah, that's going to work. Give me back permission to go and run it. Everybody's happy. You guys get more, the scenario, get more capacity. And I get what I want without having to wait. So is microservices going to impact operations in a way? Because what you're getting at is more of a microservices. More of the primitives are going to be in the ops side. So there's a development mindset anyway. Is that standard dev ops now is ops? Well, you need to handle the operational concerns as early in the life cycle as possible. Meaning developers have to build from, it's kind of like in the car world, you build a car for manufacturability. You have to build the services for operability. And so that's number one. And with the new kind of microservices decoupled world, you have to move to this model of operations. Because the old monolithic work balances across these silos, it just doesn't work in this decoupled world. It makes everything kind of grind to, back to a lump in mass of who knows what. So if you want to let the organization decouple, you have to be able to decouple your operations to match that. So how long is it taking for customers to kind of realize the value of your solutions that you bring to the table? And how much time is it saving them? Yeah, I mean, for Rundex specifically, because it doesn't force you to learn new languages, you can start with what you've got today. So literally, it's days, right? Start plugging in things that you have. You can set up the access control. You can set up kind of the options, the interface. The next thing you know, I've got this self-service interface and I can turn around and let somebody use it. So Rundex doesn't do the culture and the organization change for you. It just becomes a tool that greases that. It makes it a lot easier to get that to get that done. What specifically in the tool that works for customers that's resonating in your tool? What's the big impact when people engage with you guys? When do they know when to bring you in for the tool? Sure. Let's just say that the guru's coming in. Hey, here's the culture.  You come in, what do you do? So for us, we're kind of, I guess, more the bottom up, right? It's usually a team that says, hey, we're getting overrun with these requests.  There's too many ways to do things across all these tools. And then it's, hey, these people keep bugging us to do this. Or that team keeps bugging us to refresh this environment. Or this team, we need to give them access and something goes wrong in production to run some health checks to see what's happening. So really those kind of operational support type of use cases is generally at the team level being brought in to solve these different problems. And then where really the gas gets poured on is when the upper management is following all the DevOps and SRE conversations and realizes things need to change. Then they usually see Rundeck ask, ooh, we can use that, right? That's going to help us unlock things. And let's do more of that. And it spreads from team to team. So you're really not trying to come in and boil the ocean over.  I mean, we get organizations that aren't touching their culture at all. It's literally just, we're doing things, the old classic off-shored, kind of application operations call center model. And we're just going to get better at that and use Rundeck to create more capacity, standardize things, bring some more people into this process. And that's it. And they're very successful as that. And then, but the really exciting ones is when the code gets caught up into the larger organizational transformation. You mentioned SRE, site reliability engineers. Google used that term. So I got to ask you, we talked before we came on camera about NoOps, you know, having a NoOps culture because DevOps is more developer. And we were kind of poo-pooing that and you were kind of more aggressive. I won't say, well, you said to me, because, you know, children's show here. Yes, I'm sure a lot of children are watching the queue. The ops guy. No, pun intended. So Google is really hardcore on this. Do you have an opinion on this? Ops, NoOps, DevOps, roll-of-ops. Ops happens, right?   How many of your companies would still be in business tomorrow, or in a week, or in a month, or in a year? And people's hands kind of going, going too much, you know, a day in a week, you know? And the reality is operations happens, right? And these are complex, you know, moving systems, interacting with complex things in the world. And you have to be able to operate them. So, you know, the original NoOps idea was, oh, I don't want to have a separate thing called operations. I want to distribute to operations where it makes sense to have engineers everywhere. Google has an interesting view, which is, no, we have a distinct organization, but they call it SRE, and they use more software engineering discipline to do. They have a whole kind of methodology behind it, but they're very much proving you can still have a separate engineering and operations organization and do it right. And then there's folks like Netflix and, you know, in Amazon or more like, no, no, we're going to distribute it within these cross-functional teams and new organizations. I mean, it's still, no matter how you slice that, but here's the thing.  So just because you're automating something doesn't mean it goes away, right?  Or it can make it worse. Yeah, so talk about that pull, push, that tug between that, because it's attention that's positive, because you want to automate things that you're doing multiple repetitive tasks on, but you just eliminate some tasks, but you're still operating. Talk about that, Daniel. Well, I mean, there's certain things computers are very good at. Repetitive, knowing tasks, computers are great at. When it takes human creativity or sort of the super kind of complex connecting the dots, humans are good at that. So how do you automate as much of the things as possible that the computers are good at? And that gives you the time and the cognitive bandwidth to focus on the creative things, creative in building things, creative in, oh crap, we got to solve this, right? And the tool should be there to support that. The idea that you can automate all of that away, it just, it's not, it's just not that well. Give an example of that. If you look forward five years, I think about how we're moving fast with the evolution of the cloud, everything else that's happening, Kubernetes, IoT, AI, all this great stuff's happening. You got blockchain, you got crypto, and a lot of things going on that are super positive that it also could be detrimental. Where does the human piece come in? Where will always be the pieces where human creativity, human intuition, human judgment, where is it always going to shine? What specific things do you see never going away? I think that's what you said, the intuition and the judgment, right? In the day-to-day work activities, you need to use that intuition and judgment to get things done, to see the different signals and understand what they mean, to create new solutions on how to kind of solve these new challenges. That is where the human beings are needed. So it's both in the delivery time and in the idea of operations. You think of like an airplane, right? You know, there's still pilots. You think of a nuclear power plant. There's still operators. Tons of automation, tons of alarms, tons of things to assist them, but it still comes down to the things that human brains are good at. So there's always a role for- So categorically, obviously security, latency is the one, multi-cloud workload movement, is the areas that you start to see the categorical areas that are never going to go away, right? Yeah, and at a certain point, you're going to things where the platforms get better, and you kind of climb the stack. And more things that only human being could do in the past, you can start to use, you know, to automate things. Like deployment. Deployment used to be human task. Now we start to standardize things, have standard parts, have virtualization, now the cloud, now, you know, now all the cloud-native technologies, that allows you to say, okay, we standardize these things and built the right tooling. Now we can focus the humans on more important problems and move at a higher velocity with better quality. Great. Awesome. Well, I mean, great stuff. Okay, what's going on for you? What are you up to now these days? What events are you going to? What are you working on? What are the cool things you're excited about right now? Cool and excited about. The DevOps Enterprise Summit, you know, I've been involved that number of years. That is the best collection of, I think, enterprise, you know, big corporation thinking around a whole kind of sphere of transformation. And it's growing too, so just, right? Yeah. Give a quick plug-on. Yeah, yeah, that's growing. It's because there's one now, there's one in London, there's one now, it can be in Las Vegas. You know, 1,000 to 2,000 people. I thought SREcon, SRE is the, it's like a specialized implementation of all this DevOps thinking. I think that's another great place to be. And then, you know, DevOps days, velocity, all the kind of traditional conferences in these. Great community, got to say, being involved in the DevOps on day one, watching the pioneers, and a few with arrows in their back, but now going mainstream, it's super exciting. I think Kubernetes brings, you know, that mainstream highlights everything. That's that platform, let's talk about that. A lot of the concerns that human beings had to struggle with on a day-to-day basis are now being put into the orchestration and scheduling and, you know, the containerization of things. Producing, producing. David, great work. Congratulations on all the work you've done. Been a real contribution to the industry. Oh, thank you. And good luck with the business. Thanks for coming on theCUBE. Yeah, thanks for having me. All right, this CUBE live coverage here, Mountain View for Cisco's DevNet Create. Really the Cisco's foray into cloud native, really getting at that DevOps culture once solving big problems, programming the network. Cisco's bringing that together with their communities. Of course, CUBE's here covering it. More live coverage after this short break.


What's going on everybody? Hi, I'm Sean here late sports guys for one box This one's gonna be for Marti a one box of 2023 plates and patches football Mr. Martin a little bit. I'm talking about the back family size the family size of a sour patch kids Oh, man, and the whole thing. Yeah, not the wrong with some sour patch kids Yeah Let's say a little bit sugar in the system, but a lot of nothing wrong with that Thank you Andrew grabbing up an order as well drew Thank you Andrew. Let's go through repatches here Three plates. We have a Roshan Johnson on the road. Roshan Roshan Johnson Thank you Andrew grabbing a prison box in some purse or some group break spots. Nice. Appreciate you sir Uh-oh For the dolphins we got a swag champs at 25 with a sick number patch that one is Tyreek Hill The cheetah on the swag champ like that with the sick number patch. That's like a throwback dolphins look to it I think that's one of the throwback jerseys Swag champ To 25 Nice one here, we have a steel signatures auto of Fred Warner One of the best middle linebackers in the game out of San Francisco This one there 22 100 one of the best linebackers to do it Well Fred Warner Very nice one there. We have another plate auto. It's gonna be Jalen Hyatt Rookie for the Giants that one is number to 300 on the base auto out of Tennessee Giants rookie We have on the short print it's gonna be a will Anderson, Jr. Nice one there captain for the Texans That one is five out of five Very nice one there the captain for Texans out of Bama dog. He is a dog Nice one there my man. So good great box here Very nice end with the will Anderson to five Alrighty there that will do it for you. Mr. Mark some really nice hits out of pleats get those right out. Yes, sir


How are we doing? Welcome back to the channel. Welcome back to another video. My name is Jacob and As you've probably seen by the title today is all about updating anyone out there wondering where I'm at and what my my chest looks like six years and in two days time two months on from the day that I went under the knife Under local anesthetic and got to my memory glands dissected and removed from my body Which is something that I've always wanted Did I know that the recovery was going to go as terribly as I did absolutely not but no one does and looking back now I'm super glad that I did vlog of that experience because it has helped a hell of a lot of people out there and I continue to get comments and questions and All that are viewed of course on my content around that subject and so today Because I've got a comment recently asking, you know, what does it look like six years later? It kind of made got me thinking, you know, but why don't I? Next time I'm in the gym bust the camera out and try and give you guys a Realistic look at what it looks like Now my recovery was was terrible But I made certain mistakes along the way the first of which was taking the drains out too early because I was interstate and I had to travel back home or at least I didn't have to but I wanted to I want to get back home as soon as possible gave myself three days to recover jumped on a plane and The rest is history by day six or seven. I developed massive hematomas They stuck around for a long time But After about a year and a half or two years The discoloration underneath the skin finally went away and now six years on there is no discoloration But in saying that number one my nipples are completely numb and Number two they if I if I brush something really hard against them They are super sensitive and really sore Really sore now, you know, I can lay on the carpets or lay on my stomach in bed and you know Run and jump and play sports and do all that kind of shit without any pain whatsoever But if something brushes them hard enough and like the wrong way It really really hurts.  I've just finished an arms workout and I thought you know what six years on Let's do a bit of an update. So That's exactly what I'm going to do now because I think I mean I'm kind of I'm kind of confused as To exactly what type of content I should I should bring to the channel I'm kind of between, you know between video series and I'm focusing a lot of my attention on my other channel, which is cool. It's okay to talk Feel free to go and subscribe. The link is down in the description below I'm actually like that channel is about mental health. It's about addiction and it's also about challenging challenging myself specifically 30 day challenges and at the moment, I'm on day 27 of a Challenge where I've counted my calories every single day for 30 days. So I'm on day 27 of that and I've shaped up and so that's another reason why I feel good enough to make this video because I am in better shape than I was 30 days ago And I'm in a hell of a lot better shape than I was 1696 months a year ago. In fact a year ago. I was balls deep in a vicious cycle of alcohol addiction and so for the last three months at least if not longer I've been battling Battling physically and mentally to get my body back to get my health back And to feel within you know a good enough headspace to finally start creating content on this channel again So here we are This is it. Good to be able to smile.   No one should have to go through that. I think the biggest barrier to for a lot of people The biggest barrier to get in the surgery is the money and I feel like you will get what you pay for I feel like you definitely going to want to go to a surgeon locally so that you can recover Well at home and also so that you can go back to him if you have any complications such as I did So by all means guys look back in my video library. I've got an entire playlist titled gyno surgery or gonna come ask you There's information videos.  I mean a surgeon a surgeon it should be qualified should be experienced enough to not make that mistake But, you know, the fatty or The least obvious the gland is going to be so Those are my thoughts and my thoughts alone But it was about time I did this like I really couldn't be happy with the The result now six years on and six years is just gone like that And I mean I could have quite easily put the surgery off and put it off and put it off and still be dealing with these these Mindfucks that's having gyno doesn't matter how big or small your cases will give you as a man We're not meant to have breasts. Okay. I still haven't taken any steroids for my entire life I'm 33 if I get to 40 And I still haven't I think that'll be a miracle I feel like I'm getting closer and closer to just, you know experimenting with a small dose and just seeing, you know, how that feels as a man with physiological levels of testosterone and that I Will bring to the channel. I'll bring that that that experience to the channel. I feel like you guys Will be interested and I'll certainly be interested good bad or ugly. We'll see what happens I haven't done it yet, and I don't have any plans on it in the immediate future But like I said, I get to 40 and I haven't done it. I think that's a miracle. Thanks for watching guys Welcome back to the channel And I'll see you the next one