It's going to be really, really bad.  Oh, yeah. OK. OK, so Derek, you're ruining it. OK. You ready? You mic'd up? I think so. OK, so we had a talk scheduled that was features, features, functions, kind of the same one we always do. We decided to blow that up and do something different. So what we're here today to do is to, Chris, stole all of my architect thunder. So I had all these great slides. And I was like, Chris did them all. I'm going to do something different. And so what we did is we're going to do a demo live on stage on Terrible Conference Wi-Fi, although you have a hard line. OK. Hope it works. We hope it works. We're going to do a live demo of OpenShift 4. And we're actually going to go into why we think this is so important. So Chris kind of talked, we're going to show. So Mike, you want to kick us off? Thanks. Next slide there. So everybody look to your left and then look to your right. I seriously want you to remember when you first saw OpenShift 4, because this is going to be incredible what Derek is going to show you today. We're calling this the mother of all demos. You're going to want to build a personal memory around this.  As I said, we're going to show it to you. We're going to get through these slides as quickly as we can. But what built it? Why did we make the decisions we made? If you hit the click here, was it the over 1,000 customers that we now have on the Kubernetes platform that we know as OpenShift? The next slide, or the next? Or was it the 32,000 support issues that we took from the customer base? It was because of the 32,000 support issues. Next one, could it have been the over 17,000 commits we gave back to the Kubernetes open source community? Most of those from the 32,000 support issues. Next slide. Maybe it was the over 500 professional services engagements we do every year to bring our customers to production on Kubernetes. That's the other 15,000 issues. Hit it again. Could have been the 122 releases now of Kubernetes that we have given to the market, or the $250 million we just spent of Clayton's bonus on buying CoreOS. I like to say a quarter of a billion dollars. It sounds more exciting. Or the 4,000 pods per day that get deployed on OpenShift.com, on our starter, our pro, and our dedicated hosted services. So again, we're going to go at the fast of these slides. Honestly, it doesn't matter if you're here and you don't believe in Kubernetes. I don't know why you're here, other than to just be a hater or something like that. Kubernetes was a big multiplier for application operations. We talk about serverless. We talk about DevOps. We talk about all of these things. The end goal is to change how we do software to make it easier to be faster, to be more responsive. As Chris pointed out, it's to make things automated. So Kubernetes has evolved. OpenShift has evolved with it. It's been about building applications and running applications. It's not about computers. We're not, computers are just a thing. It's like the electricity that runs through the wires. It's about software. And software is what we care about and what we're trying to help build because software runs everything. I'm pretty sure that none of us would have made it here without software. We probably wouldn't have been able to get in the door. The doors are probably tied up to some crappy industrial IoT thing that would have just failed and locked us out of the conference center. So if Kubernetes was a multiplier for applications, our goal with OpenShift 4 is to be a multiplier for Kubernetes. And so continuing, we were involved very early in the community. It was about stabilizing Kubernetes, making it enterprise, which various people will tell you that's the worst thing in the world to make something enterprise, which means it actually works and it's probably secure. Although last week's CVE, even then, this is the reality of the world we live in, right? Which is, there's so much software. I get a little, maybe this is a little too down, but I'm gonna say we may never be safe again unless we ourselves internalize this as we have to get a lot better at patching and rolling out software updates. And that's part of that CoreOS DNA. So a lot of our mindset is how do we do Kubernetes in a way that we can build the future on top of it, which means when CVEs come out, you need those fixes before somebody else gets to them. As you know, we brought CoreOS in and we released Container Linux at Summit, has Red Hat CoreOS. It really is a foundational component of this new stack in that it allows us to have an immutable, align to an immutable infrastructure that is extremely lightweight. It means no more snowflakes. It means that we can leverage a more impactful orchestration for you that we sort of were blind to before. And we're gonna see that today in the demo. And so Chris really hit on automated operations, but I don't know that I would have picked such a dystopic picture to put up there. Like refineries are often like the bad things we talk about, but we talk about operators, automated operations. Anybody in this room who believes that we won't need to operate software, please raise your hands. Yeah, I didn't think so. So going forward, software has to be run and managed. What's gonna do it? Well, we could do it by hand, which is what we've been doing pretty much since the beginning, or we can get better at automating it, which we've been doing since the beginning. So this is just a natural evolution of where we're going. CoreOS put gave it a really good name. Thank you to Reza and Brandon and Alex. Like the name of an operator is it's about people and it's about computers working together. It's not just one or the other. So that was all the slides. So now drum roll please for the first time ever. Yeah, yeah, yeah, yeah. We're gonna do a live demo on stage of OpenShift 4. And all of the things we talked about over the last year at Summit in the open source communities, we're going to show you how we're going to take these pieces and put them together. And it's probably gonna break along the way, but that's why we do live demos, because it's exciting. Well, thanks for that vote of confidence. You're gonna do great. So far, so good. Yeah, so I think I'm probably the man and the platform here is gonna be the dog. Hopefully it doesn't bite me. Woof, yeah. So the demo we're gonna do today is in three parts. So the first part here is gonna talk about your day one experience. Since the merger of CoreOS and Red Hat, we've kind of reevaluated how we want our initial experience to be to deploy the platform. And along with that, we've gone and built a new installer. And so this is gonna be a bit like a cooking show. And so I'm gonna try to put a cluster in the oven. And hopefully a few minutes later, one will come out. So the new install is designed to help users from a wide range of skill levels, like the novice user all the way up to the advanced user. And what I'm gonna walk through here is the very novice user. And then we'll talk about the advanced user afterwards. So we have a new tool, OpenShift install. And it's got a few commands.  You can find information dependency graphs about how this installer works. Dependency graphs, really? That's because some of our friends at CoreOS love graphs. So let's explore this create cluster command. So what can I do? The most naive user is gonna come here and say, well, what I really want is a cluster. So let's go and do that now. And it's gonna ask me some information. This is a guided wizard-like experience. So it's gonna say, okay, well, you want to go and provision a cluster on some infrastructure. Tell me who the SSHK. And you can pick up and down through. And then you need to- This is the demo, guys. It's the CLI interface. We know everybody in this room loves CLI interface. Yeah, it's amazing. So the next question, you see the top level question it's gonna ask me. It's gonna say, well, what's my domain? And now I have to type a few things. So hopefully this goes well. And I will name my cluster commonsday. And now it gets to an interesting question. It's asking, what platform do I want to install in? Right now, the experience we're gonna walk through is using RedHack Horos on AWS. And we'll grow the installer to support additional platforms over time. So make sure I'm on the right thing. And because I chose AWS, it's now gonna ask me, well, where do you want to install? And so- Don't pick London. We'll probably lose a transatlantic line if you do that. So this is actually installing in our developer accounts. And hopefully I don't hit quota issues. So US East 2 is gonna be awesome. And this is gonna go great. And like that, that's all it did, right? So now the cluster in a few minutes is gonna start provisioning. But basically, it's gonna pick some appropriate defaults and start provisioning infrastructure and create the cluster on my behalf.   It's not that we don't actually care about Terraform, it's just a detail, right? Like we're trying to focus this experience, what we're showing, on ensuring that you succeed in installing a cluster every time. And some of the choices that you may be used to, historically, are maybe things that you shouldn't do right up front, you should do later. So we'll get to that as we go. Right, so I didn't provision any computers here to use the novice installer flow. It's doing it all for me in the background. So while that's cooking, let's start looking at the more advanced path. So the new install tool works a bit like make. So it has a series of targets that get executed in a particular graph flow. And depending on where the operator wants to come in and tweak their install, they can do just that. So rather than just go direct to create a cluster, let's look at the other things that we can create. So that wizard you saw where it asked me a few basic questions to figure out where I wanted to install the cluster and how. That's creating what we call our install config. So let's step through that and take a look at what that is. Okay, so we're gonna go through the same flow again and give it the key. Uh-oh, typo. It's important to get that base domain right. Is it gonna fluster you if I sit here and point out all your typos? Yes. Okay, so rather than what you saw previously I answered my questions in the wizard and out was a cluster being cooked. This didn't go and cook a cluster yet. What it instead did was go and create an install config file. And I'm gonna bridge what I show here so you don't see some data if this is going on the internet. It's a secret. Yes. So this install config file is just a YAML file, right? And so the information that was collected in that wizard you'll see here as well as some initial defaults that were chosen. So behind the scenes, because I chose to install in AWS in a particular region it's going to create me three masters and by default the installer is going to provision those masters in the set of availability zones exposed in that region. And then the default cluster size for your worker node pool is three replicas. Now I could come in as a more advanced operator and tweak this install config. So I could say I want workers and GPU node types for some other set of machine platforms instead of just master and worker. I could set some default sizes for them so I don't just have three workers, I could have 10, 20, 30. And then depending on the platform that I install in this platform section could let me choose more particular metadata about that host environment I'm installing too. For example, the instance types that might be used. So once I tweak this install config and get it as I want the next step in the install flow is to create some Kubernetes manifests. I would also say at this point like I bet you there's a lot of people who look at this and they say, wow, no, there's like 7,000 other things that I also configure. Part of what our goal is, is many of those 7,000 other things actually aren't things that you should have to deal with. Sometimes they are, sometimes they're things we want to tweak. We work with a lot of sophisticated customers who need tuning parameter X and tuning parameter Y and they need to drill down into these things. One of our key principles and philosophies with OpenShift 4 is don't give you that choice until you actually need it. We want people to be able to script installations and to do the crazy complex stuff if they need to. But upfront we want you to succeed first, tweak later. Yep, and so part two we'll talk through some of that day two operational stuff. So, a lot of people might be familiar with how Tectonic and CoroS's distributed working in the past and present. And they basically self-host a Kubernetes control plane. So in OpenShift 4 we're doing the same basic actions but we self-host a little differently. But the way the installer works is it takes that install config and then creates some Kubernetes manifests. Let's say how the cluster should be applied. So it read that install config YAML file and now it's created some manifests that are literally just regular old Kubernetes manifests into directories. So I have some manifests for some default namespaces that'll appear in the cluster. Default manifests for some config maps and various information like that. And I could come in after the fact and tweak some of these things if I wanted to change how the default queue manifests work for that corresponding install config plane. Now the final piece to how the install works is once you have that install config that's now been translated into Kubernetes artifacts which are just deployments and secrets and namespaces, et cetera, is to create a set of ignition configurations. So if you are familiar with ignition can you raise your hand in the room? Okay, so not everyone is familiar with ignition. If you were a CoroS container Linux user ignition is a agent that runs in the operating system very early in boot and it can read from a remote location to figure out how to configure that immutable host before being fully booted up. I like this. I would say ignition is cloud in it that doesn't suck but since we're being recorded I wouldn't say what I would actually say to that. And part of ignition is we wanna connect machines to a central brain wherever they are and it's not really about the cloud it's about the software that runs on that cloud, right? So you still have to run an operating system somewhere whether it's you running it for somebody else or somebody else running that operating system for you and ignition is that bridge between a machine and a cluster. So remember the goal here at the end of the day is to get a cluster that's running an immutable operating system. So for Red Hat CoroS you need to be able to tell how to configure that operating system on first boot and collect that configuration from ignition. So basically that previous cooking show you saw where I created a cluster that wizard went right through the defaults and went and created some initial ignition configurations that tell how to bootstrap the cluster. So we'll do that here and we'll take a look. And this ignition configuration step has gone and read the Kubernetes manifests that were created previously and actually I missed something, my last step. And that manifest flow besides creating the Kubernetes artifacts it also created some default TLS pairs for how certain components in the system were allowed to interact securely by default. That security that we talked about. We actually, you know, thinking about this is we know a lot of, this is like a big point of feedback is probably of those 32,000 support issues I think something like 31,000 of them were my certificates expired. And so one of the things we're trying to do from the very beginning with OpenShift 4 is not just at the cluster level, not just at the application level, not just at your ingress level but start building out that mindset where you actually have a cluster that can rotate certificates for you because every time a certificate doesn't get rotated a puppy loses its wings. So we wanna get to that glorious future of no one ever gets a certificate expired warning at 9 a.m. on a Friday because they set their alarm for daylight savings time versus regular time. With that in mind, so the installer here at the end of the day to bootstrap the install process it's basically taking these three ignition files that tell various nodes in the cluster how to boot and how to configure yourself automatically. Because we run a self-hosted control plane you kinda have a chicken and egg scenario, right? We need a Kubernetes cluster in order to run a Kubernetes cluster. We overcome that chicken and egg scenario by having our bootstrapping machine. And so that's what this bootstrap ignition file is. And just so you guys can see it's not too complicated. And another thing is like, the point of all of this is that once you have this running is you don't have to go back and reinstall it. And so a lot of what we're trying to do is to build in that idea of if you can change it live this was I think something that Tectonic did really well. If you can change it live you can recover from issues, you can fix problems, you have feedback. You don't have to go reinstall those clusters. And while there are a lot of people out there that would love to have immutable clusters I think based on the evidence a lot of us are gonna have clusters that they're gonna wanna run for five or 10 or 15 years. And so some of this is building in early the foundation of if you can't manage it on the cluster if you can't change it after the fact it's not worth doing. Yep, so just to demystify ignition it's just serving a JSON payload that's read by the machine on early boot. And if you look through any ignition file it's basically saying these are the list of default units I wanna apply. These are the list of files and where I want them located in what on what file system. So this install process that we walked through was able to create enough of a baseline initial ignition configuration to boot up a cluster. And so time to JSON for this demo was about, what was that, 10 minutes. So we promise from here on out we're gonna be a lot less heavy on the YAML and the JSON even though I'm sure there's people in the room who really love that stuff. Okay, so previously a cluster was cooking we'll see where this is going at it's still cooking. Ignore the times it really doesn't take this long even though it says it's gonna wait 30 minutes. And the way the installer works is it goes in by default we'll provision a bootstrap machine and a set of masters. So I asked for three masters it's gonna go and provision me three masters. That bootstrap machine boots reads that concrete ignition configuration that we created earlier and then knows how to serve up ignition configurations to other nodes in the cluster, right? So that bootstrapping machine will serve the ignition config for the master as well as every worker that comes up. From that bootstrapping machine we know how to go and create a XED cluster and a temporary control plane to then know how to install a real control plane on the masters. And then we get to a point where we just install some operators in the cluster and the cluster becomes self-aware, self-managing and we can tear down this bootstrap machine. Does that mean that the entire cluster is now made out of operators? Yes. That leads me into part two. So the install once you run it you never use it again to do an upgrade. Basically the entire cluster now is self-aware, self-managing and entirely automated. So if you're an existing V3 OpenShift user and you were using OpenShift Ansible to both provision and upgrade a cluster, the V4 experience now is basically you use the install to install a cluster and then you depend on Kubernetes native applications like operators to now automatically upgrade and deploy that cluster. So just in summary there you've given us a single command line install experience for Kubernetes. Yes. You've given me a template that I can get as pretty sophisticated with myself if I want it to enhance it. And the thing comes up by itself. I don't have to do anything. It'll be done. And you've separated the tool set from the upgrade tool set. Now it's install and an upgrade. Everyone should be super thrilled on the app. Oh my goodness. Quick, keep holding the time until the cluster comes out. So anyway, the cluster we're baking it'll take about, I don't know, 10 minutes to 13 minutes in that AWS dev account. So hopefully later in the cooking show we'll see that cluster. So let's get to part two, which is I have a cluster. How do I operate this thing? So we talked earlier about operators. Operators really just fancy term for automating things. Kubernetes does a great job in managing and deploying your own applications. But there's really no reason why Kubernetes can't do a great job to manage and automate its own deployment of itself. Where Kubernetes was in v1 versus where Kubernetes is now in v1.13, the world has evolved. And Kubernetes can now support the right primitives for us to actually use it to drive itself. And I think this is actually, I want to take a second there, is Kubernetes 1.0 was a very simple thing. We'll talk about some of the keynotes this week about how Kubernetes has changed over the last four and changed years, four and a half years. It took a long time to get to the point where we believe that all the pieces are in place in the community and the ecosystem to where we can do these kinds of demos. And that work is work the Red Haters and Googlers and CoreOS people and people from VMware and Amazon and 1,000 other companies have contributed to. So this is a commons meeting. I wanted to pause and say that statement that Derek made. It took us a while to get here because big important things take a while to happen. I know that some of you have gone through those 32,000 support issues, and I'm sorry. Part of our apology to you is to come up here today and say that that was actually worth something that's going to make the next five years easier. So in the interest of time, Clayton's going to need to talk a little less and let me get to my demo. So this is the admin console. I'm logged in as a temporary admin user. And this is the console you'll see by default after the install finishes. The console is deployed and you can log in, give you your credentials, and bam, you're ready to go. The critical operator I want to talk about today that drives how OpenShift's distribution Kubernetes can self-automate itself is what we call the cluster version operator. That is the top-level operator that knows how to manage everything else. And so if we click through, and I'll show a little less command-lining it a little more visual here, is I can look at my cluster. And this is reading what we have, which is a cluster version resource. And it's telling me what my current version of my cluster is. This is literally running master in production in a demo. So my current version is what was deployed. And so hopefully, like I said earlier, my previous creation of a cluster just finished. And from that, I can see an update source. And it's telling me, hey, potentially updates are available. Tell me what I want to go to apply. And I could roll out a new upgrade of my entire cluster just by applying a new version of this cluster version operator, which will then know how to coordinate the upgrade of everything else. The cluster version operator is itself just a currentized deployment. And we're going to get a little geeky here. So Derek, I didn't have to log into the R&J network and figure out if there was a new update of. No, you didn't have to do that. Come on. This idea of delivering updates to the cluster, this is the big trick. It's kind of the magic trick that is the cluster manages itself and knows how to pull updates. It knows how to safely apply them. That's just reconciliation. So if I went in and deleted half the things on Derek's cluster, that operator is going to be like, I can handle this. I got it. And it'll bring you back to where you were. Kubernetes is about reconciliation. You say what you want, and we make it happen. That infuses everything that we're doing in OpenShift 4. So I messed up my environment right before demoing. And so it's telling me I messed something up. And the top level operator is trying to apply something that it's not applying. Don't worry, this doesn't always happen. But like I said, I was running master right beforehand. But when I look at it. It stops safely. That's the important part. Yes. And it's telling you something about it. So if I describe the cluster version resource, it's telling me how it's trying to converge to my desired state. I want to run this level of OpenShift. And it's making a series of coordinating changes to the cluster and trying to converge to that desired state. At the end of the day, it's managing a set of second level operators, we call them, which are basically managing the Kube API server, the controller manager, the OpenShift API server, et cetera, et cetera. And if any one of those operators is having trouble converging to its desired state, that gets bubbled up very clearly to the admin that says, hey, there's a problem. Right now, I have a transient problem on one of my masters, so please ignore that for the moment. The way the cluster version operator works is it takes a release payload. So a release payload in OpenShift v4 is just a container image that knows how to start the cluster version operator and then has a set of manifests that know how to install every other operator. So we'll take a look at that. And you can see there's a payload that says, this is the image I'm running that's defining what this cluster is. And we have a new command called OCADMReleaseInfo, which I can feed at that payload and I can find out how that cluster was installed. And hopefully. Come on, conference wireless. So you can see here, this is telling me, OK, you're running a 4.0 payload of a particular version.  All the cluster version operator that it's doing is just applying the corresponding operators for those components into the cluster. I can get a little more even detailed on that and say, hey, what commits am I running? And my zoom is really big, but you can see my operators are from various components within our Git repos and the individual versions they're running. Excuse me. So like I said earlier, the top level operator we'd run is the cluster version operator. It's just the deployment that's running on the Kubernetes control plane itself. And all it's doing is it's acting like a replica set, deployment a stateful set controller. And it's just constantly trying to converge to a desired state. And so it's not doing anything too crazy. We can actually go look at it and LRSH into that container. And when I say we use Kubernetes to manage Kubernetes, this is the set of YAML artifacts that describe the resources we deploy that say, how you make this version of Kubernetes so. So for every operator we ship, which includes all the regular suspects you'd expect, an operator that can manage the network, an operator that can manage DNS, an operator that can manage certs, the API server, the scheduler, yada, yada, yada. Basically, this release payload that's inside that image is telling the version operator what set of artifacts you could cut or apply to the cluster and converge to where you want to get to to do an upgrade. So I want to do real quick. So time checked, Diane. I know we're running a little bit late. How much flexibility do we have? And does everybody in the audience want to keep seeing this exciting demo? We're going. We're going. Just speed it up. Can everybody live without coffee for like an hour? Can we bring coffee around for everybody? OK. I'll try to go quick. So the point being we have a top-level operator that just applies cube artifacts. So in the same way that you roll out new versions of your applications on the platform, that's literally how we roll out new versions of Kubernetes. And Kubernetes is really good at that type of thing. So if I go and look at the logs for the cluster version operator, it's not like you're doing an upgrade. We're always doing an upgrade. We're always trying to ensure that if the cluster has drifted out of your desired state that it is getting put back into line. So just like the man was getting bit by that dog, the cluster version operator is stopping me from going and changing one of these things. I can't go set a flag on the Kube API server. It will come back and say, hey, don't do that. I'm going to go set it as it should be, unless I go through a formal interface for it. So that's the cluster version operator in a nutshell. And we've gone and spent a lot of time going and building a set of second-level operators that manage the core control plane. Everything. Everything on the platform is literally an operator. From the CNI plug-in to the storage plug-ins to the Kubernetes API server, literally everything. And for the last year, an entire year, 30 scrum teams have been porting the Kubernetes framework into operators and baking in their knowledge of what those framework components should behave. And I'm not going to lie.  And so some of these are doing the same things you might do in a control loop as an operator. You run a control loop and you say, here's what I want to have happen. And I'm just going to do that every 30 minutes until the end of time with a cron job if you like to live it dangerously. And so this mindset, again, it's all about just making sure that we know what's happening, we react to it, and we test the living daylights out of that from the first second it's installed on the cluster. So when OpenShift v3 first released right when Kubernetes 1.0 came out, the state of Kubernetes at that time and the state of where OpenShift v3 needed to be required us to package and deploy Kubernetes slightly differently. So you had one Uber start master command or one Uber start controllers command that combined your Kube processes with your OpenShift processes. But then it let us go and do things like RBAC and some other things. Kubernetes has evolved. We spent a lot of time at Red Hat trying to change the upstream to make it that we can get clear layers between our software and the upstream. So what you'll see in v4 here is that we actually deploy Kube API server like any other pod in the cluster. We deploy the scheduler like any other pod in the cluster and same with the controller manager. So the base control plane itself for Kubernetes is just running as pods. So you can see them here. And then there's an operator that's actually managing that to ensure it's where I want it to be. What's cool about these operators relative to existing tectonic users is we've kind of re-looked at how that worked and made it that we don't need something like boot cube recover in a disaster scenario. Basically, this thing is always going to run fine disaster or not, which was really cool innovation from the team. Then the traditional OpenShift API server stuff that gives you all that nice development pieces is just running as a separate operator as a daemon set on every one of my masters. At the end of the day, an operator is just managing a set of custom resource definitions that describe how the admin wants to make their cluster be configured. We have a lot of custom resources. And so for a lot of our operators, like you'll see, we have a DNS resource, an ingress resource. The admin interface by which you manipulate OpenShift E4 is just tweaking these custom resource definitions. And then depending on those operators to converge the cluster to your desired state. Yeah, we've made configuring the cluster just another Kubernetes API operation. So you can say what you want the state to be. You can run kubectl apply. You can put it into Ansible. You can put it into Helm. Doesn't matter. Configuration is managed as a declarative API, just like everything else. OK. So who wants to see what Red Hat CoreOS is looking like? No one. No one is excited to see what Red Hat CoreOS looks like. That's the coffee. They're running out of coffee. So right now I'm running a six node cluster, three masters, three workers. And what I'm going to do here is going to paint some of Red Hat CoreOS developers. I'm going to SSH into a master. I promise in the future they say they're going to taint the node when I do that. But right now it's not there. And so you can see right now that the OS that's being rewarded is that I'm running Red Hat CoreOS for. So if I go in SSH into one of my masters, let's take a look at what is actually on this host. So you can see I'm in and I'm running Red Hat CoreOS for. Awesome. So to clarify any confusion, Red Hat CoreOS is a rel kernel with rel content. So if I go and look at what I'm running, I'm running a rel kernel. Hopefully I can say it again. I'm running a rel kernel. So what we've gone and taken the rel atomic work and we've taken the ignition work from Container Linux and got ignition running on a rel kernel. So on first boot, in order to configure this machine, ignition reached out to its home server and said, how should I configure?  I'm very sorry. Nobody changed him, OK. You misspelled successfully. Well, I can't type. Either way, I promise if I had this journal command, you would see that on first boot ignition launched before the whole OS was booted and said, hey, tell me the files I need to lay down on this immutable host to configure that host. And it would have been good. Now, just like rel atomic or Container Linux, I am running an immutable OS. So I'm an apharius user. And I'm going to go and try to touch bad. And this should tell me I can't do that. But I do have a writable layer. And I will go and try to touch. Good. But I have no permission. That's also good. Security folks. SC Linux is on and enforcing by default on Red Hat Core OS. That's Dan Walsh in the back clapping. We're just pandering to the crowd at this point. That took a lot of work from the team to get that working with Ignition well. So congratulations. Unlike rel atomic and unlike Container Linux, we are not running the KubeLit in a container. We're running it as just a native process on the host. And the KubeLit, I mean, the reason is that we believe that the KubeLit and the container run time in the kernel. We've talked about this before. Those are things that have a machine config. And because this is really hard to type accurately. So we're running pretty long on time, Derek. All right. Anyway, point being, there's a set of files that I can configure essentially on the cluster that says how you get there. Now, that's the operators. If Clinton didn't talk so much, this would have gone great. Sorry. So this was the set of second level operators. Also, in v4, we've been working on making sure that not just we can explore an operator pattern, that the applications that you run on the cluster can also be delivered as a set of operators. So you'll see there's an operators tab here. And this is our OLM operator lifecycle manager component that allows you to, as an ISV or as an administrative cluster itself, offer cluster services that other users can subscribe to and deploy in their projects to do things like, if I want to get a managed .cd in my cluster, I can go and interface with the operator lifecycle management component, deploy .cd. And now I can interface with .cd using verbs, using nouns like .cd cluster, .cd backup, .cd restore. Because we're short on time, I'm going to cut that. Awesome. It works. You don't have to worry about it. Go to the final part of the demo, which was machine management. So in v4, we've adopted, there's an upstream project that's been sponsored out of SIG cluster lifecycle called the cluster API component. And that has an interesting set of primitives for dealing with machines. So what you'll see in the administration section here, you'll see some new interfaces. You'll see that there's a machine interface. So what this machine interface is, is this is actually a Kubernetes resource called machine. And it is the base atom for describing a Kubernetes machine. By default, the installer used the machine CRD definitions and this other component that knows how to then instantiate those machines in your target platform to stripe out my cluster. By default, because I was deploying in AWS in a particular region, you'll notice that everything is spread across in the right number of availability zones. This machine interface, as I said, a machine is the base primitive. But just like pods, you have a machine set. And so I'm able to say, OK, in US East 2A, 2B, and 2C, I want a particular number of machines. And if I wanted to grow that machines, as a day two operation, I can just say, all right, well, I want to spend a lot of money and get 1,000 machines. Click Save, please, right now. Save. No, I'm going to do this on demand, so it'll hopefully be cooler. I don't think we have quota for 1,000. As I said earlier, a machine resource is just a YAML definition. This is being derived out of the upstream, and we're adopting this portion of the project and we're really excited to work with the upstream community to see this through fruition. So what I want to do is I don't want to spend a lot of money unless I have to run something that needs it. So like I said earlier, everything in OpenShift, and I mean, literally everything is managed by an operator, including an autoscaler. So I'm going to turn on autoscaling for this cluster. And it's a set of operations I can configure for the autoscaler, but I'll just go with the defaults. We don't have a nice, pretty UI for this yet, sorry. And so by deploying a cluster autoscaler CRD, I've now enabled the cluster autoscaler component. And what I'm going to do, and excuse me while I do this quickly, is tell those machine sets that they're allowed to scale. So by default, they resize one each. And what I'm going to do is deploy a machine autoscaler resource that's going to say, for each of those three machine sets, which are targeted to particular zones, that they're allowed to scale between one and 12 compute nodes on demand. And so as workloads come onto the cluster, Kubernetes literally itself is going to see that, hey, I don't have enough available compute to satisfy the workload. And I'm going to dynamically size up and down my machine set on demand. So let's do that now. I have a job I'm going to deploy to deploy a work queue. And as we said, we don't really think that managing machines is interesting. We think putting machines to work is interesting, and this reinforces that. It's we have an API for machines. That's the last thing you should ever have to think about machines for. They're up to date. They're secure. They're running workloads. That's the goal. And programming the infrastructure to do that for you is, again, something that dog, the man doesn't have to go and say, I want 10, or I want 15, or I want 20. The machine says, I know how to talk to the cloud API. I know how to ask for more machines. Make it happen. OK. So I went and deployed a work queue job that's going to run, I think, 100 parallel pods that make absurdly high requests for the actual work they're doing and require me to spend money to spin up new machines. So there's not enough capacity on this cluster to satisfy this compute need. But because I went and told the cluster autoscaler and the machine set API that says we're allowed to dynamically size these things, in the background, if everything went well, instead of having six machines, I now have about 30, right? Or a little slowly increase. And this is just on demand, creating compute. Yeah, and this demo is on AWS, but we don't really think AWS is special. Oh, I'm sorry. We don't think AWS is any more special than the other clouds or OpenStack or on-premise clouds or, honestly, bare metal. So Chris talked about bare metal, but we don't think that this experience is unique to clouds. We think this is just something that should just work everywhere, and that's part of what OpenStick is about. The way this machine API works is the primitive for defining a machine is agnostic of the platform it's running on. And then there's a very specific provider section that says, OK, if I'm running on AWS or if I was running on Azure or any other bare metal, you can give an environment-specific configuration. And there's a very tiny component called an actuator that knows how to create and deprovision machines on demand. So all we've done right here is done that with AWS. These machines, when they come up, it takes about three minutes for the machine to appear and then be ready in a schedule workload. So I know that we're tough on time. Way over time. But we need a time auto scaler, we're so far. So you can see that the nodes have booted. They're going to start to go ready. And the workload will be satisfied. And the pods will finish running. The cluster will size down. I'll stop spending absurd amounts of money. And in the end, that is managing compute on OpenSHA4. And that will be the configuration of the tool. So there's a ton more in this demo. And you're not going to be able to see it here if you find Derek. Make him do this demo for you. But then we were like, well, this is really easy for us to go do as a development team. So what we said was, well, why don't we just make it available for everybody? So you had a heart attack. Yeah. And this man, I want to say thank you to this man for not having a heart attack when we sprung this on him. But you can go to try.openshift.com. You can do everything that we just did in this demo. Get Pulsei Grids, get access to OpenSHA4. We'll walk you through the process. Derek's demo is up in a Google Group forum. And this is just the beginning. We said we want to be that 10x multiplier. We want your feedback, commons, contributors, and companies. We want to make sure that the way that we're going is a direction that makes Kubernetes and building applications and running applications at scale easier for you. So please give it a try and let us know how we're doing. Thank you. Thanks. Thank you.


All right, and this is the network defense room in case any of you guys are lost. All right, I'll wait a couple minutes to let everyone filter on in. All right, I'll go ahead and get started. Drew Hintz. I'm going to be presenting on COVID channels and TCP and IP headers. My email address is up there. Feel free to email me with any questions. My website is up there pretty easy to remove. It's go.new, like the pronunciation for GNU. So go ahead and get started. A brief overview of what I'm going to be talking about today. First of all, who are you? Why might you have a need for COVID channels now or at some point in the future? Who are they? Who are your adversaries? Who are you trying to avoid by using them? I'm going to give a few quick definitions, talk about a couple different classifications of COVID channels. Then I'm going to go into an analysis of a few COVID channels in TCP and IP headers. Then I'm going to talk briefly about an attack against a timestamp COVID channel and how you can detect that it's being used. Then I'm going to talk briefly about detecting and preventing COVID channels, because if you're someone who didn't want this hidden path of communication in your network. And then I'm going to go ahead and talk a little bit about a tool that I'm going to be releasing today that lets you use in Linux, use some of these COVID channels to actually communicate and send data. So first of all, who are you? Why might you actually care about COVID channels? Why might you actually use them at some point? So you say, well, I need to communicate securely. I'll just encrypt everything. You just sort of apply cryptography. Everything's great. But sometimes that's not the complete solution. Maybe encryption is either outlawed or maybe there's mandatory key of screw. And if you don't know key of screw is it's where you have to give a third party to give someone a copy of your private keys so that they can encrypt your information if they have to do an investigation of some sort. There was some talk of this in the U.S. with Clipper and Fortes and all sorts of fun stuff. In the U.K., even a few years ago, voted down some mandatory key of screw stuff. But even so did this day. If, for example, the police were to capture your hardware, they could subpoena your keys from you in order to make you turn them over so that they could see what was on there. And also in other situations and other form of covert channels, you might not want to let anyone know that you're communicating with someone. For example, if you're in jail and if you're in China and you want to communicate with some known dissidents and you knew the government was watching these dissidents and looking for who they were communicating with or who they wanted to watch, then you probably wouldn't want to let them know that you wanted to communicate with them. Other things, your employer might not want you going to certain websites or might not want you talking to certain people.  You wouldn't want to let people know that you were the one controlling it, so you'd want to obscure your path of communication with them somehow. And so now, what are you up against?  Maybe someone who's trying to watch everyone. Someone who has some sort of IDS. Maybe this is your network admin at work. Maybe it's the government watching everyone's traffic out there who knows. But probably the system that they would use, some sort of automated system that would look for keywords, similar to maybe a signature-based intrusion detection system, something like that, looking for sort of weird behavior traffic that was easy to detect. In this sort of scenario, they would probably only be able to keep a very minimal state on what was going on. They might be able to keep state on your individual TCP connection, but not your past 1000 TCP connections and probably not even that much information on you individually. However, maybe you're up against a dedicated observer. Maybe someone is honing out on you and they're watching you specifically. Maybe as an example before, you all want to use dissidents that's being watched by a government or something and they want to see everything that you're doing according to every packet that you're sending out, every bit, and they're analyzing as much as they can that a team of people are looking at it, and you still want to be able to devade them. To devade them. In this sort of scenario, they have a lot of resources and they can watch everything. Going along with the two ideas or with the idea of what you're up against is how much protection you actually need. You could perhaps probably be semi-covert and try to fool the casual observer. You could use unusual traffic that if they looked at it, they would notice it was weird, but hopefully they're not going to look at it. Hopefully they're going to look at what's pre-dying and some real secure that they go and buy. And doing this would require just a load of moderate amount of work on the behalf of someone trying to find out if you're using this channel. And so now it's really categorically or pretty really difficult for anyone to detect that you're using it. You might be using, even if they've read all the cover channel papers out there and had implementations of all the things, it can be pretty much theoretical underlying cryptography below it. And hopefully they can't do that and say they could even break the cryptography. They would have to know where to look for the cryptography to throw it through the big cracking machines and try to crack it. So all of this, so wait, what is it? It's dropping a microphone. Alright, so... Can you guys still hear me? No. Probably because it got turned off. Is that better? No? Something like that? Does that mean better? Better? Alright. So what is the cover channel exactly? I'm talking about why you might need them, but what are they? So kind of two ways that I've kind of classified them is communicating extra information to a host when you're allowed to communicate with that host. Say you're allowed to go to... you're allowed to talk to your friend Bob, but everything has to be in plain text and people are allowed to watch you, but you want to send them something secret.  This would be examples like if you were controlling a group of distributed trojans or something like that and you didn't want anyone to be able to trace you back to that communication. So now first we'll get extra communication to a host. This is when you want to hide the fact, perhaps, that you're encrypting data, because if you're allowed to communicate with them you could just send them the encrypted data, but let's say you're not allowed to. If you don't want Kyosco to come into play or it's out water would raise them in the eyebrows. So basically what you do is you send them some sort of normal data, like you'd normally send Bob some FTP file or something, but then you subtly alter it or you subtly alter parts of it so that there's some sort of covert message going outbound in the traffic. The receiver then grabs the traffic as normal, it looks like normal traffic, you can even use it as he normally would and then analyzes the traffic and then pulls the message out of there. So in this case, if someone were looking at it, hopefully a casual observer or some sort of observer would just see a normal communication going on, some sort of daily thing you were doing anyways, so they wouldn't think anything of it. This draws a lot of ties to secondography, kind of the same sort of thing. Secondography is simply the art of hiding information or something like that. You send someone, a classic example, send someone a JPEG or something and then you would simply modulate your message in the low bits of the JPEG so that it wouldn't actually alter the picture that you were seeing, at least significantly, but your message would still be encoded in there. So how could you find a good covert channel that you actually want to use in this manner for attacking on extra information? In everything that we do, there's always some sort of randomness that's going on and you need to try to find this randomness and then replace it with your own carefully chosen randomness. For example, some random places you could find initial sequence numbers. If they're good and secure against hijacking TCP spoofing and all that stuff, then they should be generated randomly or at least pretty randomly. Also, complex timing of network transmissions. It'd be pretty hard to analyze and see exactly when your kernel's going to spew out certain packets or when it's going to do things and say you could use that as a source of randomness that you want to replace with your own data. And in this, the random data that you see, you're placing with your own quote random data. And the random data, you can call it your own random data because it's encrypted and a good property of encrypted data is that it's random. Release with typical encryption. So a simple example that I've kind of described. Say you're Alice and you want to send the secret message to Bob. Say periodically, say you just went on a vacation or something and you took a whole bunch of pictures and so you want to send them to Bob. So you hop on Bob's FTP server and you send them the pictures and Bob gets them and looks at them. That's what sort of an attacker might see. But let's say I'm also subtly in there is that for each TCP segment that you threw out there, that you threw to Bob's FTP server is that you start encoding your hidden message in the padding of the TCP header. And I observe we're looking at this, hopefully we wouldn't notice that the padding wasn't zero as it normally is in a communication but it was actually some sort of data. But Bob would know, hopefully he already communicated to him to record all this traffic and to look at it and then he would know to look for the secret message hidden in the padding. And so now let's talk a little bit about unseen path of communication, kind of untraceability if you want to. And you use this when you don't want your association with the node to be known. You don't want it to be known for example things you've already mentioned if you want to communicate with a closely scrutinized node someone that you don't really want to be associated with if you're going to be accessing forbidden material for example say you're in some country that considers certain things dissident like if you were in China during certain periods when they considered CNN.com to be dissent material and so they blocked it at their firewalls and they wouldn't want you to access it. Let's say you wanted to access it now even though they still might consider it dissident then you would want to be able to not show that there's any communication between you and CNN. And of course you could also use this for malicious activity and preventing law enforcement from tracing you back or whatever you would want. Not that I'm advocating it or anything. And in order to do this you pretty much use another node or another set of nodes to relay information for you. Perhaps these nodes know that they're relaying information for you perhaps they've been set up beforehand and purposely designed to relay information for you in a good and secure manner. So how would you look for a channel like this or what would you look for in a good channel something like this? If you're using a node to forward your traffic on you would want to make that node pretty hard to monitor for example say you were say like a real life example or something say you were someone in the US and you want to communicate with some mafia boss in the US you might send your message off to someone in Romania who would then hold on to it for a while where perhaps the police aren't as vigilant in Romania they're not watching as much stuff so they wouldn't be watching this guy's mail then he would take it and eventually send it back to some mafia boss in the US and if the US was just able to watch the US mail then they wouldn't notice that you sent some message to the mafia boss and so in this case the node would be harder to monitor and that the US government would be much harder for them to go and monitor this Romanian node another attribute you would want is that you have to kind of trust this node they've been using you trust that this person in Romania isn't going to go blab out to the US government that they were forwarding messages for you and that's something that you need to be concerned with in the electronic world also because trust is even harder to establish there and it is possible though even if all networks are being watched to actually afford yourself from protection not necessarily if all nodes are corrupt that's another story but if everything is being watched there are methods that you could do this using mixes and basically a mix is where you have a whole bunch of say middlemen or Romanian people you send it off to one of them and they forward it on to someone else but each time they get a letter from you they also provide the service to a lot of people so they receive a lot of different letters from people perhaps they build up and they wait until they have 100 letters in their hand then they take those letters they mix them up and then they send them out in a random order and if everyone were watching that node they would see that you communicated with this person this middleman, this mix and along with many other people and then eventually this mix communicated with 100 other people and so they would be able to say someone in this 100 person group communicated with someone else in this 100 target group and if you had several layers of this it would grow exponentially as to your space of different possible communications if you just had five layers you would be able to communicate with each layer and you would get pretty complex pretty fast another idea or another method that's out there by the way the mix thing this sort of thing has been implemented with re-mailers such as Mix Minion and I think Roger presented here I guess yesterday they did before talking about Mix Minion so hopefully I went to that and you know what I'm talking about already and I'm just boring you but onion writing is another method of encryption to wrap your data in multiple layers and that's where the onion term comes from and so you basically get a list say you have five nodes that you want to relay your information for the final node that's going to get the data you first encrypt it with their public key so only they can open it then you go up one node before that encrypt it with that node's public key and so on and so forth until you have sort of these five shells of encryption in there and each node can only remove the graph to that node that has the key for the outer most shell they then rip it off and then they forward it onto the next person and so on and so forth and no one can actually no one that gets it early on can tell what data you're transmitting or who your eventual target is whereas people towards the end are very far away from you as the originator of the information so they can't trace it back if they're corrupt and there's an actual project out there called the onion writing project that started at some naval research labs at some point it's going on and also on top of sort of using nodes just to forward information for you you could also use some of the earlier methods of earlier classification to obscure the fact that this node was even getting information to pass on or passing on information in any respect so an example of this we kind of talked about a few real world ones this is kind of a timing one so say Alice and Bob are allowed to make both make requests to some web server say they're in some country where they're heavily scrutinized and Alice doesn't want it to be known that she's communicating with Bob they say they're both allowed to access some local news site which doesn't have very much bandwidth the small server is not very powerful so what Alice does is whenever she wants to transmit a Wonder Bob she makes tons and tons of requests to the web server hits it really hard so that this performance will degrade that web server, he would notice that it lagged a lot or several requests would notice that it tended to have a high latency and he slowed this deployment to some information and whenever Alice wanted to transmit a zero she would simply not make heavy requests to this web server and then whenever Bob made a request to this web server it would come back much faster and of course this assumes that Alice and Bob have enough bandwidth to significantly alter the state of the web server and Alice really like doing it to CNN.com but you might do some little local server and it's close to you so now if you have a covert channel what are the sort of properties that you want to look at what do you actually want to evaluate in it first of all you probably want to look at the bandwidth because that's kind of the whole purpose of it is to transmit information in a certain amount of time in a certain amount of things unless you have a bandwidth in terms of bits per second or something like that connection or bits per packet things like that another thing you want to look at is ease of detection we've already talked about this some with how easy to detect how much prevention do you need and there's always sort of a tradeoff here between bandwidth and how covert you're going to be another thing is permissibility how often will it actually go through a network if you start sending out illegal stuff stuff that's just random and weird sometimes networks will filter it or rewrite it and destroy the covert channel another thing is how hard is it to prevent say this stuff starts becoming commonplace and everyone starts using it how hard would it be for network administrators how costly would it be for them to restrict these covert channels to limit their bandwidth significantly so that they became unusable another issue is how hard is it to implement you have all these great ideas and all these covert channels which people have known about but up until this point there haven't really been good implementations that kind of brought all of them together in a usable package except for some of the end-to-end communication where you obscure the path you're actually communicating and what special cases or restrictions are there that are out there that you have to keep in mind when you're using the channel so that you don't misuse it and get yourself killed or something so what are we going to look at today I'm going to look at extra communication covert channels because they might be useful in case encryption is restricted or outlawed and there's a key subpoena by some court somewhere I'm not going to look too much at hidden path communication because there's been a lot of good work in there with onion routing mixes and things like that and we're going to focus on TCP IP headers because they're pretty universal TCP IP is flowing out all around there and also by just using the headers you're able to piggyback onto legitimate track as an FTP example you don't have to somehow magically make up pictures to send simply use traffic that you're sending anyways and piggyback on top of that in the header so the first one to look at a very simple example is using the TCP urgent pointer and the urgent pointer is a thing is a part of the TCP header there are kind of two components to it one is the TCP urgent pointer control bit and right there next to synanac and fin and this says hey look there's an urgent pointer that you should pay attention to what the urgent pointer itself is is it's a sequence number which points to just beyond the last bit of urgent data and this isn't actually used too often anymore or maybe not at all but the urgent pointer is always supposed to be interpreted if that urgent control bit is set so what you could do is simply not set the urgent control bit yet stuffed it into the urgent pointer pretty easy to do however someone might you know one thing you could also keep in mind is to actually the point to some data so it has to be somewhere near the sequence numbers that you're using so perhaps you want to make sure that the urgent pointer could be pointing or could be near data and sequence numbers that are actually TCP session so now let's look at some of the properties of it it's going to exercise really good bandwidth for a covered channel 16 bits per TCP segment might not sound like a lot but it's quite a bit for a covered channel it could be much less if you try to restrict what the urgent pointer could be so it looks a little bit more reasonable how hard is this to detect it's pretty easy to detect the urgent pointer is rarely used and it should especially never be set or used if the control bit isn't set it's just supposed to be all zeroes and not even supposed to pay attention to and also you could start to look at and make sure the urgent pointer actually is somewhere pointing to some sequence data if you actually started setting the urgent control bit to try to make it appear a little bit more stealthy how hard would this be to prevent? moderately easy you can make it moderately easy if you just examine each urgent bit each urgent control bit whenever there was a line you write the urgent pointer so that it was all nulls if you want to just flat out and rewrite it that would just make it easier a few other things for miscibility currently this will pass through pretty much everything some firewalls and normalizers will of course look at the urgent pointer and see that the control bit isn't set see that there's something anomalous going on we'll rewrite it for you back to all zeroes and it's gone how hard is this to implement it's really easy to stuff bits in there as it's going out in a special case you of course can't use the urgent pointer when you're using the urgent pointer legitimately but this really doesn't happen too often going along with the urgent pointer similar things are starting to use the padding and observed bits similar to the urgent example and that they start to break protocol and they start to do things that no TCP stack would ever do they'll lower bandwidth simply because they're typically fewer bits but you could of course combine this with the urgent pointer to one thing and use that the padding would be easy to detect because if someone said zeroes just as reserve bits would be really easy to detect is misbehaving and some routers may start to drop packets if the bits are set because they just see them and they see that they're just bogus and they should never be like that and sometimes they also rewrite the padding for you especially if they start messing around with some of the options and adding another option is just keeping on that another code channel that you could use in the header is just using the IP type of service the IP type of service in case you're not familiar with it there's a quality of service that you're requesting from the network that your data is going to be flowing out on different things that have like precedence delay, throughput, reliability and then of course they have some reserve bits so all you do is you just set the type of service byte to your data and set it on through but if you want to be a little bit more stealthy the delay bit in the type of service is actually used by say SSH whenever you're sending over keystrokes so if someone were to shift through it they couldn't just look for delay bits being set because they're actually used legitimate so the bandwidth of this if you use the entire type of service field then it's one by pripy data gram which is pretty good if you don't and you only use the delay bit or you only maybe use the delay the reliability bit or something like that then you would get one bit or two bits or however many bits of the type of service field you actually wanted to use and it actually would be really easy because in the entire type of service field because you're never supposed to use it all especially the reserve bits and very rarely are most of them actually used especially the precedence ones however if you use the delay bit it becomes moderately difficult to actually detect because someone couldn't simply shift through and look for delay bits being set they'd have to keep track of you they'd have to look for you using a lot of delay bits in your transmissions perhaps in non-standard ports the telegrams typically didn't use the delay bits and they could also look at it and see that if you're just stuffing encrypted data directly into the delay bit that you somehow had a perfectly even distribution of delay bits being set and delay bits being not be set because in this case you're taking some not very random data and replacing it with really random data and so that'd be something that would be detectable also, permissibility it'll pass through pretty much everything especially if you just set the delay bit because that's something that is used and people pass on all the time prevention it would be pretty easy to prevent you could simply just write out all the type of service bits to zero whenever any packet came through you statically this could slightly alter how you're actually providing service but no one would probably notice no one would probably complain how hard would this be to implement it'd be really easy to get into the type of service field and you're done special cases once again could slightly alter the traffic but probably not noticeably or at least not enough to care about prototype the initial sequence number in TCP sequence numbers are used as an index into the data that TCP is transmitting there's a unique sequence number for every bit of data that's being transmitted out there and from some earlier security the initial sequence number should be randomly chosen to prevent TCP hijacking or TCP spoofing so you can't pretend that you're actually getting packets being sent back to a host and so you want to choose your initial sequence numbers to be the message that you want to transmit so normally your kernel or your TCP stack or whatever would somehow randomly choose a random value to throw and to use for the initial sequence number so all you have to do is go in there and replace that function with your function that transmits the data for you and stuff your data into that field and have the receiver get it so bandwidth, this is pretty low it's only 32 bits per TCP connection of course this is really depending on what sort of traffic your piggybacking is on if you're piggybacking it on one huge FTP transmission you're only going to get 32 bytes or maybe 64 however many if you're just in an FTP session you generally don't make that many separate TCP connections but say we piggybacked on a web server request where you hit the page and you downloaded maybe 40 images you typically make separate TCP sessions for each of those 40 images and all of a sudden you'd have 40 times 32 bits just for that one request which isn't too bad detection for something like this would be you know, quote impossible in that they would expect purely random data to be coming for initial sequence numbers and you'd be giving it purely random data so just by looking at the network they wouldn't be able to tell preventing this would be fairly difficult they'd have to start proxying all of your TCP connections and keeping state on every TCP connection which some proxies serve as old data but that's more sort of a local thing rather than permissibility typically passed on through except for some proxies like I said difficulty of implementing pretty easy you just go and grab that function replace it with the one that sends your data special cases some operating systems like Windows 95 definitely don't transmit random initial sequence numbers they do sequential ones, yay what a thought but as long as you're using a real operating system like some Unix brainer then you should be pretty fine with doing that there was even I think on stash that a couple weeks ago someone put a paper up in a study analyzing the randomness in different initial sequence number generations and ended up working on improving the ones in Linux and I think the BSD ones came out the best another technique is to modulate the TCP time stamp options a little bit and the TCP time stamp is simply just a TCP option that marks the time that this packet was sent out and this method was presented earlier this spring had privacy enhancing technologies 2002 which was right before CFP and at low bandwidth the low bit on a TCP time stamp is going to be random so if you were just communicating with someone the time stamps that were flowing out through you if you had a silicon action like sam modem or something they'd appear pretty much random if you had a high speed connection though they're definitely not going to be random because the time stamp counter is incremented every millisecond or something like that and so if you're sending a packet faster than once per millisecond then you're going to see lots of repeats so the bandwidth of this sort of transmission would be pretty low one bit per TCP segment not too bad detection for this sort of thing would be very difficult if you were on a low bandwidth connection because the bits are pretty random to begin with and granted you're placing them with really random data for the things you could do to prevent that prevention from an administrator's point of view it would be pretty easy to do this you could pretty much just disallow the TCP time stamp option and rewrite the header and just get rid of it permissibility pretty much everyone seems to allow TCP time stamps through how hard does the speed implement it doesn't sound too difficult but there are a few subtle things to keep in mind you're twiddling with the low bit the TCP time stamp and just like time so your time stamp should be monotonically increasing so in order to do this say your time stamp ended in a one but you wanted to flip it down to a zero you're kind of jumping back in time so instead of what you have to do is you have to wait until the time stamp rolls up to a zero which in Linux is referred to as one jiffy so you have to wait one jiffy for the time stamp to roll up and doing that you delay the packets you delay that one packet by one jiffy in which case if you're on a high connection the packets would build up because this packet would be delayed as would all the ones behind it they would also be delayed if you're on a slow connection you wouldn't have that problem because you probably wouldn't have any packets sort of backed up waiting to go that would actually be affected by that so how could you detect the time stamp on a fast connection when you're on a fast connection you're going to have to the covert channel will slow the transmission of the TCP segments down to a fixed rate and it's fixed or converges to a fixed rate because you're assuming that you're transmitting uniformly ones and zeros because the transmitted information which doesn't seem to have any sort of pattern to it so an algorithm you could use to detect this is to look at a host and start counting the number of different values of time stamps that he sends in a certain amount of time and while you're doing that also count the total number of time stamps so essentially counting sort of the number of times that he repeats time stamps but he doesn't repeat time stamps and you take the ratio of the total time stamps that were used over the different time stamps that were used and you get a ratio and the ratio if you're using this covert channel are modified will converge to about 1.94 that's not from any math or anything just from actually going out there and doing it so how could you actually and if you're doing on a fast connection like it's as fast pretty fast as you could get on hosting the networking it'll reach up to about a ratio of 20 or something like that however if naturally your ratio is less than 1.94 then it's not going to be slowed down at all and therefore it becomes you can't use this method to detect it so how could you prevent someone from detecting you if you're using this method on a fast network well since it's slowing it down to a sort of fixed rate how would you slow it down a little bit more and that little bit more to get a somewhat random value so now you're slowing it down by a random amount maybe every connection maybe every once in a while you change it to sort of model your computer might be busy or something so it might be being a little bit slower than normally would and doing this prevents the attack of course it would still be detectable that you are slowing it down and that you're slowing down your transmission which would be something fairly marvelous to begin with but it could happen if your server so also how could you detect the timestamp cover channel on a very slow connection so in a slow connection the low bits appear pretty much random as I said earlier however when you eject this encrypted data into the low bit it becomes more random than it normally is so what you basically have to do is look for this increase in randomness in doing this so what you could do is just get a whole bunch of TCP time just record all bunch of TCP timestamp low bits and then put it through a randomness test the test to see whether it was truly random or not truly random but really random as much as they can tell and then quite a few tests on the internet you can just go and download and throw this up and if the program comes back and spits it back and says yes these bits are really random then the cover channel is most likely being used however if it comes back and spits it back and says these bits aren't very random and you can probably guess that they're not using the cover channel based on this unmodified form and so in order to prevent this you could try looking at transmitting some data that wouldn't appear to be completely random by using some sort of encoding schemes and so now detection and prevention of cover channels in general so first of all detection for some of the semi-covered stuff try to just detect anomalous traffic so it has some of this stuff like invalid checksums and things like that sort of working a little bit on just historic preprock that would look for bad things like bad reserve bits and bad things in combination try to correlate some of them together and keep state TCP sessions how could you prevent this? you could perform normalization in your traffic if you think normalization is basically getting traffic in from somewhere taking it doing something to it to make it the way you think it should look use often when people are worried about 5-out or something evading their IDSs so they want to take it and defy and all their data, desegment all their data so they get to decide how the overlaps are handled and all that sort of stuff and you could apply this in pretty much the same way by taking it and stripping out type of service base, taking it and stripping out urgent base, things like that but you know your network doesn't really need and there's some programs out there that can do this Norm is one, I think it's written by Vern Paxson but we're not sure BSD's packet filter and then there's also a more advanced method like with the timing channel stuff if someone was attempting to communicate by using timing like there was a whole bunch of requests and then not a few not many and then a whole bunch and then not many you could use a pump method basically your router or something collects a whole bunch of data for a while and then sends it out in a burst kind of like it fills it up and then empties out the queue right away granted this costs a lot of resources but it can help to prevent that sort of stuff that's more of a theoretical type thing and one thing to keep in mind is that you can't close all covert channels if you're letting someone communicate data back and forth they can get covert data through all you can hope to do is decrease the bandwidth and increase the difficulty of actually using covert channels to the point where it becomes infeasible so now a few implementation issues if you actually were to try to go out and implement these covert channels well you need to have encryption in there at some level otherwise the covert randomness that you're stuffing back in aren't going to be very random you need to have somewhere telling whether the data is transmitted or not you need to be able to index into the data and say what data is being transmitted and then of course you have to consider the age of the issue of reliability versus bandwidth so first of all encryption is the key component you have to use good encryption to ensure that all these channels are secure if your encryption is flawed somehow say you're doing something similar like some shift cipher then you're using the initial sequence number covert channel then someone could look at this sequence numbers and tell that they weren't very random see some sort of patterns emerging in them and see if something is going on and also if you want to use a secure covert channel you need encryption because you have to assume that your covert channel is well known I mean everything I'm talking about today is well known it's been published in papers and discussed for ages and I'm just kind of presenting it here so you have to assume that your attacker or the people monitoring you also of course know about it and probably have teams of people working on it and another sort of side issue with encryption that you have to keep in mind is you have to ensure that you don't transmit the same ciphertext over and over again like say you just set up some sort of public key encryption system and you were just encrypting the data you were sending out with the recipients public key say you send out the same message repeatedly a few times message like hi do you want to go get lunch today or something like that but you send it out every couple of days and there was no time stamp or anything in there like that and so it was always the same message and if you were sending out with the sequence number method then they would see maybe they would see this repetition of initial sequence numbers happen and they would know if something was going on so one sort of thing you could do to this is use some sort of cipher maybe a stream cipher or something and in addition to using a key with that you could somehow add in a common time that you both share maybe even just the day to try to alleviate the problem of that and so now how do you tell the person that you're sending data to if the data is really being transmitted because if you're just engaging daily data conversation with this person or daily day to day community network traffic you're probably not sending covert data the entire time you probably don't have that much covert stuff to set so one thing you do is whenever you send a group of data then send a checksum at the end this also provides some reliability and things of that nature and then all the receiver would do is shift through the data look at the data and look at the next little bit of data and see if that was a valid checksum from the previous data in this case you have to make sure that the checksum is securely keyed because otherwise an attacker could do the same exact thing shift through the data look for the checksum and find out if there was some sort of covert communication going on another thing you could do there's some sort of magic flag that the receiver watches for sending encrypted and the same thing and the receiver would examine the traffic repeatedly and look for this sort of maybe it's like a sin essentially I'm going to set up a connection and everything another thing you do is just constantly transmit covert data and so they get the same data over and over again which wouldn't make much sense for binaries or things like that but her simple text messages and the receiver wouldn't know I'm getting the same message over and over again but they really just send it once so now how can you tell the receiver what data is actually being transmitted you need to have some sort of index into the data because you're probably not going to be able to send your entire covert message in one packet or one segment or one whatever so one thing you could do is just share with the data sequentially just go through and send it out bit by bit and hopefully they get all this sequence, they get everything and they start listening at the same time that you start sending however another solution you could use is take an unmodified portion of the header and send the data and use it as a nonce and take this nonce and then take a hash of it and then modulate that hash with the total size of whatever the data you're transmitting to get an index into that data and transmit data beginning at that index and that works but the problem with that is that you don't get to choose all of the bits that you're transmitting so you have to sort of transmit essentially random bits out of the data that you're sending and eventually you'll probably transmit all of the bits that will probably knock off my microphone so can you guys hear me in the back still kind of alright you could implement more advanced protocols you could have some sort of TCP over the covert channels whatever you want that provides more reliability and robustness and of course the age old debate of reliability versus bandwidth if you use a hash of the nonce you get pretty high reliability because you're sending the same data over and over again they don't have to start listening at the same time but you get much lower bandwidth because you're sending the same information over and over again however it is a pretty simple solution to implement another thing if you did the transmitting data sequentially and provide high bandwidth perhaps the highest bandwidth you could achieve but it'd be lower reliability if you're listening at the same time and get everything in the right order another thing you could do is implement a more advanced protocol that would kind of trade off some of the bandwidth for more reliability and more acknowledgement that it was actually received alright now on to the the boring stuff I guess we'll talk a little bit about the tool that I wrote this tool is just a kernel module uses proc that modifies outgoing TCP traffic by replacing the hardstar transmit function it's based off actually some code I got from the people that did the timestamp cover channel it's all GPR and everything like that the receiving component simply sniffs incoming traffic using libpcap and looks for the cover channels in the data cover channels that I've implemented in it I've implemented the initial sequence number TCP timestamp load fluctuation with high speed protection which is optional so it'll drop it down to random amounts so that you're not rid of a high speed attack the urgent pointer channel IP type of service and using TCP reserved bits and all these options you can turn them on and off but now they're just with pound defines and you have to recompile it but the interface isn't exactly the best thing and then we have to when everyone has sent data you have to cat it out to slash proc cc and it goes out there that's something I'm going to work on two different types of data indexing that I've implemented in there just straight up sequential transmission of the data takes the data encodes it in there, sends it out other type is SHA of the unmodified portions of the header and the data uses to index into the data and the SHA can be keyed and these are both options and all the options both the center and the receiver have to agree on what exact configuration you're using so they know where to look for the cover channels they know what sort of indexing you're using for things of that nature so some work that still have to be done to improve the user interface so that people might actually use it probably most people aren't willing to set up nice little shell scripts to send out their data after the device maybe make an ice gooey or something like that if anyone is looking to help with it and you know knows a bit about KD or you know programming or something just shoot me off an email I'm more than happy to get some help with that I'm going to build some encryption into it so that you can basically just send in a message and maybe give it a key or maybe you know with a symmetric key or public key whatever and then it sends it off for you so that it once again becomes actually much more usable rather than having to have your own crypto layer of it adding a few more options for more cover channels so that you can start to get even better bandwidth out of it for semi-cover communication that you might use if you're just worried about someone spotting what's going on also it would be nice to go out there and analyze how a variety of different routers and IDS's handler detect this sometimes a legal data like setting the reserved bits maybe write a little program that would go out there and send it out bad traffic like set a reserved bits and see what the other one got and then do that in sort of a manner and then report back to the user exactly what sort of cover channels they can use in that link and have a few little scripts that kind of do it but they're not very polished in any way and maybe build sort of a database of different sort of typical ones that work in typical backlands that you can pass them through in different normalizers that have to treat it and also it would be good to implement some sort of more robust protocol for the data transmission that handles indexing and reliability and all that sort of stuff something similar to TCP or something like that so you can go ahead and jump on the web as soon as I actually get an internet connection the wireless connection has been flaky have people been able to get it to work today is there something magical you have to do like some magic word or something you have to say I made a funny cap and caps I tried that and it's still taking that pity bit I'm lazy and before I let you guys all go kind of a shameless plug the open source vulnerability database is a project that's just starting up and it's going to be a vulnerability database that's run by the community that contains information on publicly known vulnerabilities right there in the wild different than CVE it will be more than just a naming system but it will also contain information on the vulnerabilities, perhaps exploits for them ways to detect the vulnerability what systems are affected, things like that and the website for that is www.openvisosvdb.org and it's not quite operational yet but it's going up there question? I had one question from back before you said implement the type of service you said isn't passed through all rather than possible will it actually disrupt SSH traffic? it won't actually disrupt it we'll still see that we'll still see this going on SSH tries to set the delay bit for keystrokes so that it goes through faster yeah I imagine it could, I have no idea I had no idea but it would be really cool if it could be maybe we've just got the source to windows you have the source right here it will be as soon as I get a net connection yeah, all the source and the tool to detect the TCP timestamp channel and the presentation let me know the 2018 presentation supposedly appears with some action stats cool


My name is Sonia Gonzalez, I'm the architect to an archimede farm director for the open group and I have also been working along with Bill in the harmonization project which is looking to harmonize a ton of an archimede and we have a series of white papers so I'm going to make some questions to Bill and we're going to have a chat about the project right now. Bill, please. My name is Bill Ostrom, president of Metaplexi Associates and Sonia and I have been working together for the past little over two years I think to try and make togaaf and archimede work better together from a practitioner's point of view and now we've published our first set of results and for phase one of the project and so we'll talk about things related to how well the standards work together in terms of your terminology, the modeling views and viewpoints as well as the fundamental definition of the terms that are used and the entities and their relationships. So the talk of an archimede both are open group standards, they have been developing in different paths because they are addressing different needs and concerns and as a general approach what would you suggest to final users to use the two of them successfully? Well I think you have to use the archimede language in order to describe your architectures and you do that as part of overall enterprise architecture process and so the togaaf standards are designed to allow you to have a method that you can follow as well as a framework for the content that you develop and so I think archimede provides that content and there needs to be some consistency between the two and again as you pointed out they were developed in different time frames by different people so there are some natural differences that occur but I think largely those can be overcome with some careful planning in terms of how you do the architecture modeling and how you set up your content framework. Both of them have different set of concepts and like you said togaaf is a framework so it's bigger, larger than archimede, archimede has fewer concepts but some concepts are like similar some others have some differences so for example in this case you're trying to harmonize concepts and glossaries between togaaf and archimede do you think it's worth to take just the whole togaaf glossaries in this larger to the archimede one that just takes some some of the points the touch points that both standards have? Well I think from an architecture practice point of view the more you can standardize your terminology in general the better off you're going to be the more consistency there is between the practitioners the less likely you're going to have confusion but then when you get down to the specifics of developing architecture descriptions using consistent terminology is important so yeah one of the things that we've done in project harmony is to basically look at all the glossary the terminology that's in the glossaries of both togaaf and archimede and see where they're the same and places where they're different and then suggest alternatives that might be able to help people to develop a consistent glossary in terms for their own organization I think that's the really important thing here is that each organization really needs to kind of figure that out for themselves some organizations will really do a very good job of a very detailed implementation of togaaf and they'll use archimede and great detail and others will be maybe a little bit less orthodox a little bit more casual and so the degree of harmonization that they need that might be a little bit less I think you also have to look at the operating model of the company to understand you know the level of rigor that's needed in order to standardize these terms so you can integrate different parts of the organization and make them work together yeah also every organization has some glossary so at the end it's like my glossary and I do it to harmonize that with both standards at the end of the day is that this is like with the screen delivery value to my organization on the other hand we have the content meta model from togaaf but it's like they said like a framework for making enterprise architecture so where practitioners are making or starting making enterprise architecture they got into the content meta model and they have this issue okay how I'm going to fit this set of entities and relationship for actually delivering models and deliver architecture descriptions and then on the other hand we have archimede meta model which is different even though some terms and some relationships are similar they are different than the ones in toga what kind of advice or approach we do the liver to find and use it to overcome that well I think again a lot of that depends on the organization and the level of maturity that they have and I think it also depends a lot on the types of tooling that they're using to develop the models and the repositories that are using the store of content so that there needs to be an agreement just like there needs to be a standardization of the glossaries of terms there needs to be a standardization of the entities that are used to create the artifacts that you're using to describe the architecture and then also the relationships between those entities and so that's the stuff that we've been doing with with the project harmony is looking at those relationships between the entities as well as the entities themselves and how they need to be similar or they're related because you have to harmonize glossaries then you come up with the content with the elements the entities and then you have to harmonize the relationship between them so everything is related and that comes to the last question about viewpoints I mean in toga we have a set of viewpoints some users say that it's too general to high level and also our company has a set of viewpoints which is larger than the one in toga and it's like more focused because it's like a language notation and model notation but if we're going we're trying to address stakeholder concerns what would you do with these viewpoints would you touch that point how do you solve that issue I first have to figure out the equivalency what kind of equivalency is there already between what uses for viewpoints and what is required to get sometimes the names are different and they're really describing the same things other times there's actually some semantic differences between what one the what the argument language requires and what to get provides so again it all boils back to your organization's needs so maybe able to just take the few points that you need and not worry about trying to make everything so we'd like to see we'd like to see you be able to use our commit to describe pretty much any artifact or any of the toga artifacts but we're not you know I think it's really important to remember that what we're trying to do here is make these two stands work together but we're not trying to integrate them together you know like weld them together so that they cannot be used anything else we want to see our committee useful for other other enterprise architecture frameworks and we want to see toga be able to use other your tools like you about modeling tools and things like that and also regarding viewpoints and it will depend on what your stakeholder concerns also there could be a specific viewpoints that could be a combination of several viewpoints that's always yes yeah that should always be the basis that you use for making decisions about the viewpoints so thank you very much for sharing your opinions with us and let's move forward with the second phase of the program okay thank you very much yeah thank you so much