Okay, so we can get started. Hello. Hey, so hi, everybody. Thanks for being here today. With Scott and I, we're going to tell you about building Cloud Native applications with containers, functions, and managed services. So first a little bit about ourselves, Scott. Yeah, so I'm Scott Colton. I'm based out of Sydney, Australia. I work in the developer advocacy team for Microsoft and specializing in Kubernetes and container runtimes and all those sorts of things. Been in the Docking Captains project for about four years. So probably Phil was the only one that was older than me and unfortunately bought Red Hat, so he got kicked out. So now I'm like an old GROG Docking Captain. And next is Patrick. You know, Phil cannot joke about that. Oh, no. So hi, everybody. I'm Patrick Chenezon. I work with Scott in the cloud advocacy team at Microsoft. And before that, I spent four years at Docker with Tonys that talked just before about Billkit. So Billkit Container-D, I'm a big fan here still. And so my focus is on containers on Azure. So today we're going to talk about one big topic which is over the past few years people have been using Docker and then Kubernetes to build these cloud-native applications. But nowadays more and more these applications become more complex and typically you're building an application with free components, containers, serverless functions that respond to events and manage cloud services. So in this talk, we're going to try to cover the developer experience for that. How do you package your application when it contains all these components and how do you make it scale? One other thing I wanted to say is one of the reasons why I was super happy to join Microsoft is that Microsoft's mission is around productivity. So it's about empowering every person and organization on the planet to achieve more. And it started with Bill Gates creating visual basics. So it started in developer tools. And so Microsoft always had a strong story around trying to make developers more productive. And I think with the history of cloud-native applications over the past five years, now we're getting to a stage where it's really important to start to take a look at how we can make developers more productive. So these three abstractions that I talked about, containers, functions triggered by events and managed cloud services, over the past three years, there's been a lot of portable serverless platforms that emerge on top of Kubernetes. So you have things like the FN project by Oracle, which characteristic is that it has a nice way of composing functions. So Clio, which is a super high performance and more suited for intensive data processing. OpenFast, which is like the Swiss Army knife of serverless platforms. It can do a lot of things, and it's super easy to use and get started. GalacticFog and OpenWisk. More recently, there's been two projects that are more like middleware on top of which you can build a function experience which are Knative and KDA. So we'll talk a little bit about KDA here. And the CNCF has a serverless working group. So these diagrams that show what a portable serverless offering is are coming from their white paper that I highly recommend if you're new to this space. On the Azure side, we have our own first-party service that's called Azure Functions that we manage at Microsoft. But all the runtime for it is open source. So you can run Azure Functions in containers in your own Kubernetes cluster if you want. It's all on GitHub. So I'm going to talk about the dev experience, and I nuked my slides here, but I'm going to talk through it. It's been the third time we mess it up. The dev experience I wanted to talk to you about is called Azure Dev Spaces, and it's a service that we operate in Azure and that lets you as a developer, and it's tied with the Visual Studio Code extension so that you as a developer, when you're building an application, you don't need to know anything about Kubernetes. You don't need to have the runtime of the language that you're developing in installed on your machine. You just need to have your source code and VS code. So I'll show you what it looks like. And the goal of Azure Dev Spaces is to make it easier for developers to collaborate together when they have, when they're using common services, managed cloud services for their application. So the way the experience works is that you have a VS code plugin, and there's a command line as well called easyDS, and that VS code plugin interacts with a server-side controller that you install in a namespace in your Kubernetes cluster. And so once the connection between the two is established, the VS code extension is able to prime your project. It looks at your project and it recognizes, oh, this is an OGS project. I'm going to create a Docker file for it. I'm going to create a hand chart for it. And I'm going to connect to the server-side controller that's sitting in your Kubernetes cluster. I'm going to pass all the code to it. And on the server side, we're going to do a Docker build. And actually, I wonder if they're using the new build kit functionality or not. I should talk to the team about that, because there's lots of options there. And then it's deploying it with a hand chart. And then it opens a connection for debugging. So you're in VS code. You initialize your project, and then you can start debugging in the cloud right away. And your code can leverage a series of services that are already set up in Azure. And there's an ingress controller in there also that can create URLs. Yeah, in addition to that, it can create URLs that are public so that your colleagues can test your code in a public way, or it can create a tunnel between you and between your laptop and the code in the cloud. So let me show you what it looks like. So I'm here in there. Let me first. So here, in order to play with Kubernetes, I always use docker desktop. So I have my local kubernetes cluster, which is still starting. And then here in docker desktop, I can switch context. And here I'm going to use a path aks cluster, which is a cluster in Azure that's managed by aks that I have created. And so I checked that my context is right. I'm just going to clear that out. And so I'm just going to do a kubectl get knows to show you that this is not the one on my laptop. It's the one in aks that I'm connected to. I opened it. I created it 65 days ago. I hope I won't have any problem with accounting because I should delete my cluster more often. So then I do a kubectl get namespaces. And you will see that I have installed Azure Dev Spaces in my cluster. So there's one instruction with the azure command line to do that. Once I've done that, it creates this azds namespace. And when I do kubectl get all dash n azds. So when I look at what's inside of that namespace, you will see I have a demon set to pre-poll the images that I'm going to use. And I have some services for the controller, a service for the controller, tiller that is used for Helm, and then traffic that's used as an ingress controller. So once I've done that, I can start working on my project. So I'm just going to do a code in azds node web front end. So that's a simple, that's a very simple node.js application just to show you what the experience looks like as a developer. And so in there I don't have much. I just have like a node project, package.json, and some css on javascript. So it's an index.html page. So pretty simple application. I don't have much. And I'm a web developer. I don't know anything about kubernetes. But I want to start, maybe my application is using some cloud services that I have set up. And so here what I'm doing is that I'm going to go in the command palette and I'm going to use the extension azure dev spaces, prepare configuration files for azure dev spaces. When I click there it asks me whether I want to have a public endpoint for my application. So in this case I'm going to click yes. But if it's a really secret application I could say no. And you can see what's happening there is that here I have a docker file that appears. No, I haven't installed the extension yet. So I have a docker file that appears. Some configuration for azure dev spaces that I could modify. And then a hand chart. And in VS code I have a launch configuration, which means that I can start debugging my application right away with that launch configuration. So I don't need to set up anything. And if I go in debug mode there you can see I have a launch server on azds. And if I click there you can see it's launching something. It's executing azds app with some port mapping. And what this azds app command line is going to do is it's going to connect to azure, to my AKS cluster, to the controller, azds controller on the other side. And it's going to push the code over there or sync my changes in my code. Do a docker build. And this is where there could be some interesting optimization with buildkit. And then do a ham deploy of the whole application. And then creates a tunnel between my local machine and the remote end point so that I can start debugging right away. And actually if I go... Okay, so I think this is ready. And if I go in the files, if I go in server.js in there, I can put a break point in there. And when I go in my terminal it tells me that it's available at a public end point. So if I go at that public end point I can give that to some of my colleagues and here I'm hitting the azure load balancer that goes inside of my Kubernetes cluster via the ingress controller to that URL. And inside of it I end up into my application in debug mode and so here I can... If I go in the debug view I can see all my local variables and I could modify this for example. I could say, oh no actually I need to modify it there. So I should have created a variable. But basically I can start debugging my app there. And when I say continue it's saying hello from webform and over there. So that's debugging with azds. The other experience I wanted to show you... So azds in addition to that has a few... Oh sorry. In addition to that has a few interesting features where with the URL that it creates for you you can create different branches so that one developer can have their own version of a certain part of a multi-service app running at a certain URL while the rest of the developers in the team can use the same other set of services. So it's really for teams to collaborate together on applications that use lots of services in common. Now there's another developer experience that's really cool that I wanted to show you which is called LiveShare. So VS Code LiveShare lets you collaborate with someone who has VS Code on their machine. And you can hear I'm able to give access to the code on my machine, a terminal and a host on my machine to Scott here who has nothing from this application without having him to install anything. And the beauty of it is that LiveShare works really well with azds with azure dev spaces which means that he'll be able from his machine to debug the application that we saw in my kubernetes cluster without having access to it just through LiveShare. So we're at 45, maybe I'll just skip that demo because I think it's pretty long. I'll add to the slide a link because I gave that demo at dockercon and so you can watch that demo there because it takes at least five minutes and we have just 20 minutes more and there's more stuff I wanted to show you. But just to know that both are working together. And then the third experience I wanted to show you is VS Code Remote, extension for containers. So that's something that came out very recently. It's an open source extension for VS Code. And what it's doing is that it lets you create some configuration for your VS Code project so that VS Code is split into two parts. The local part that has the themes and the UI and all the extensions and then it can run the server part of VS Code with its own set of extensions inside of a docker container which means that... And so you can use that with your local docker engine. And so what that means is that in your GitHub project if you set up that VS Code configuration with a docker file for your project, every developer who's going to join your team when they're doing a Git checkout or a Git clone of the project, they can launch in debug mode in VS Code right away. And all of them will have a similar experience. They don't need to have any runtime installed on their machine. They use docker. And their code is mounted inside of the docker container. And so you can customize all these containers. And I'm just going to show you what that looks like. So that VS Code Remote extension, that's something that was announced I think a month and a half ago. So it's six weeks old. It came out in the regular version of VS Code two weeks ago. And in addition to docker, it can do also SSH and WSL if you're on Windows. So let me close this up and I'm going to say here... So I'm in VS Code and I go in the command palette and I say remote container, open folder in container. So it's going to show me all my folders. And so I have some examples of a container-enabled projects in there. I'm going to take one where I have done the build already, so the JavaScript node 8. So when I open this one, you can see it's installing the dev container. And let's take a look at what it's doing behind the scenes. So behind the scenes, it's taking the docker file that I have in my project. It's building a docker image with it. And then it's doing a docker run with that code base inside of that docker image. And in that docker image, there's a VS Code server that's running. And plus it creates a launch.json with a debug configuration for that project, which means I can say I want to launch that program and it's going to launch my... This is a node application again. And then I can ask for a new terminal in there. And the new terminal is going to be inside of the container. So here you can see I have VS Code server running and my node process. And if I do a call of a local host on port 3000, I hit the break point that I had in there inside of my code. So here I'm debugging a Node.js application inside of a container and I didn't have anything to do to set up all that logistics. So I'm just going to say continue. And yeah, so that just works. So that's a remote extension. In terms of development, there's just one more... So these were the two experiences I wanted to show you. Let me go to the right slide. Yeah, in addition to that, when you want to debug either functions or containers in a Kubernetes cluster, there are a bunch of other projects you can take a look at. Squash is one from the glue team. There's TelePresence, which is a CNTF project. There's Ksync and Tilt, which is pretty recent. I have, if you're looking for my slides for DockerCon, I have a bunch of explanations about how to use all of them. So that covers the developer experience. Now Scott is going to talk about application packages, packaging. How do you package all these applications that have containers, functions, as well as managed cloud services? All right. Oh, yeah. Sorry. Can you hear that? Oh, yeah. All right. So Patrick's done a really good job of showing the application developer persona. But me, myself, am not really much of an application developer. I'm more of an opsy type of person. So i'm going to show you CNAP, which is an open specification from both microsoft and docker. And also, i believe hashi club and binami are now working with it. And what it actually does is it's an obstruction over tools that you already have. So these tools could be helm, these tools could be Ansible, these tools could be proper, it could be anything. And basically what it does is give you a framework to be able to Deploy these applications and code that you already have in An abstracted way. This allows you to have your Deployment pipelines really, really slim down. So instead of having multiple binaries and multiple things like that, You can have just a single way to deploy things. The best thing about CNAP is the invocation image is a Container so you don't need any binaries. You just download the invocation image and away you go. Microsoft has an implementation of CNAP called portal. What this actually gives you and why you would want to use Portal is we have built things called mixins. And what mixins are is the smarts to talk to a cloud native api. So if you want to deploy to kubernetes, you don't need to Know anything about kubernetes except the deployment mechanism That you want, the mixin will look after the smarts talking to The kubernetes api for you. Same with azura, same with Terraform and also helm. So if you've already got helm charts And you want to deploy them and pass different variables Through different yamls because you can do that with portal, Then you don't need to know anything about helm. There's a helm mixin there for you. So basically you just put the variables that you want into The code and away you go. So it's actually really, really Simple. And the good thing about CNAP, Whether you use our implementation or use docker Apps, which is docker's implementation, it's not trying To replace any tools that you've got. It's actually working with the tools you've got. And if you build a CNAP correctly, you should be able To use docker up or portal. They should be totally Interchangeable. The other good thing about CNAP bundles is there's work going on in the oci at The bottom to allow bundles to be pushed to the oci registry. So what this means is you can actually push your CNAP bundles Up with your images and the application scaffolding will Live with the container image, which is really cool. Now patrick's application, although it looks like a really Simple node application, it actually has lots and lots of Load on it. And he wants performance, Performance, performance. And if i don't give him that, Then when we do our manager's review, i'll get in trouble. No, he doesn't. A strange humor, though. But basically what we're found with kubernetes in general, And this is cloud in general, provisioning a vm is slow. So it doesn't matter if you're in azure or in alibaba cloud Or any other of the clouds. If you've got a node auto Scala, it's going to be two minutes before the kubernetes Node is ready, at least, if not more. So a project was started inside microsoft and now is in Cncf called virtual kubernetes. And what that allows you to do Is in the face to container run time and make it look like It's a node that's in the cloud. So if you're on azure, that Will be aci. If you're in alibaba cloud, I believe it's ebi. But there's a lot out there If you're on aws, it would be fargate. So what this allows you to do is actually scale your pods Across vertically without having to wait for a node to spin up. So it will hit the aci api and it will spin up a pod straight away for you. And when you're doing a highly transactional sort of workload, This can save you time and also could save you money. Virtual kubernetes doesn't give you a full node, but it does Give you enough that you can get away with a whole heap of stuff. So you can see all the different stuff there. So you get all the normal get pod stuff. And virtual kubernetes, the open source implementation, There is some limitations to networking coming back into the cluster And bits and pieces like that. If you use the implementation On azure, we call it virtual node and we'll set up the cni provider And we'll set up a whole heap of networking to speak back to your cluster. So if there's any resources in your cluster that you need to speak back to, You'll be able to do that. Alibaba cloud is taking it one step further And they've got a project called viking, which looks after dns resolution across The cloud. So at the moment we do ip across on the cni. They do dns. And they also are able to run service mesh across Virtual node, which is amazing. That was the first time I saw that was in Barcelona. But we need something to look after the scaling. And this is where keta comes in. So when a htp request comes in Through kubernetes, what we want to do is actually scale it. And we want to scale it to virtual node. We need something to look after that and look at the metrics coming into the cluster And then scale it across to the appropriate place. Which is virtual node for us or virtual kubelet. So keta looks after that. So keta can work with a whole heap of Technologies to look at metrics. So it can look at rabbit mq, Prometheus, there's a whole heap of things it can work with. And then what it does is it talks to the horizontal and Autoscaler and then talks to virtual node and scales out that way. So keta allows you to do a lot of cool things. And it runs both in a stand-alone method or it runs also in Another method which installs another product, another library By microsoft, by open deus, or deus lab, sorry, it's a habit. It's from deus labs. And it's called osiris. And what osiris does is allows a zero scale effect on a pod Because on kubernetes, if you scale a pod to zero and you have A service endpoint, once traffic goes there, like it gets Lost and there's a whole issue there. Osiris is another component that sits there and allows the Service to have no pod's belief. And then once it needs the Pods, keta will scale them up. So this is what keta looks like if you see the external Trigash law scaling pods. You've got the kubernetes Controller and all the parts of kubernetes and the horizontal Pod autoscaler so you can see it's looking after it. So keta doesn't actually replace the horizontal pod autoscaler. It actually works with it. So what we're going to do now is Just run a demo because it takes a minute. So basically what we're going to do is we're going to use Porter and we're going to install rabid mq by home. Then we're going to do a deployment and i'll go through The deployments and we're going to install a publisher and a consumer And we're going to scale it all on virtual node right now with a single command. So we're just going to put our, if i can spell, install minus c. The minus c is passing the credentials file for where my Kube control file is. So i can actually speak to kubernetes. So as you can see there it's installing the keta demo. Now i'm going to come back to this because it takes a minute For rabid mq to get up, start up. But as you can see now it's already invoking helm. So just with a single command on invoked helm and i'm going to do Some kubernetes deployment. So let's have a look at what those Deployments are and we'll come back and look at the horizontal pod autoscaler. Cool. So as you can see with that cna b bundle, What we've done is we've installed rabid mq. And then what we're going to do is we're going to do a Deployment and it's just a sample deployment. But you'll notice down the bottom here what we're going to do is We're going to say this deployment right here has a node Toleration, only the run on virtual kubelet. So that's not going to run on a normal node. It's only going to scale into virtual kubelet. Now you can see down the bottom there a virtual kubelet is ACi. If that was amazon it would be fargate or such like that. Then what we're going to do is we're going to use a custom Resource definition that keta uses to have a scaled object. And this will look and see the pollen intervals and everything That needs to happen within the rabid mq stack. And then we're just going to run the batch job, which is the Publish. So we can see things start to scale off. So as you can see there, that was a whole lot of stuff. I just played with a single command. So cna is pretty awesome. So let's go back. So rabid mq hasn't started. So what i'll do is just go to my other terminals. So as you can see here, we've just got an azure aks cluster. But you'll see that there's an extra node there that looks a little Bit different. So you can see that i've got a Virtual node running. My cluster is running 114, Which is the latest that we can run on azure. And the virtual kubelet is running there. So the virtual kubelet is acting like a single node. But what it's actually doing is talking to the ACi api. And what it's getting back, it's returning into calls that Kubernetes knows natively that a node has. So let's go back and hope that. So virtual kubelet. So we've just deployed, as you can see there, rabid mq. Now we're deploying the consumer. So if we go over... I killed that. Let's have a look. So that's going to get the horizontal auto scale off. And if you have here. So what we can see here is you Can see a whole heap of pods terminating. That's okay because they're actually running serverless Functions. But you can see there on the node That it's actually only deploying on virtual kubelet. So none of the scaling event that patrick needed is Actually happening on my AKS cluster. So performance of all the other applications i've got running There is actually nothing. Because i'm offending all this Load, this extra load that's just happened into virtual kubelet. Which is awesome. And then if i go back to my scaling, You can see now that i'm starting to get... Scaling events Happened. So you can see i had a zero replica of the consumer. And that's because i had no traffic. And as we're starting to deploy Podge, you can see the replicas are going up depending on the traffic. So before i had htb scale to zero, zero. Now i've got more traffic. I've got one. I've got four coming up now. So as you can see with that Single command through cna b, i've deployed a whole infrastructure Looking after a whole scaling event and scaling mechanism Which is quite complex. So let's go back to the slides. The good thing about this particular demo as well is i've Open sourced it. If you want the code and you want to run This yourself, there it is. It's all there. All you need is an azura account because it does need Virtual node. If you know virtual kubelet enough and you can Get virtual kubelet working on another cloud provider, just take The cnab bundles and you'll be able to deploy them there. And then there's all these other things that patrick spoke about. That's where they come. That's where you can get all the Different things. And of course down the bottom, if you Want to work for microsoft, especially in china, we're Looking for cloud developer advocates that join mine and patrick's team. So if you're looking for a new job, you love containers, And you're in china, come and see us after the talk. Any questions? Okay. Okay. If there's no questions, we'll be around outside for questions After this talk. Just a summary of what we presented Here, the goal here is when you're developing an application That has containers, functions and cloud services. There are several extensions in vs code and in azure that Let you develop that multi technology application. Cnab and the implementations. Cnab is a specification plus some implementation that lets you Deploy these applications and then keda is a pretty good way Of scaling these applications and an open source project that Runs into any kubernetes cluster. So cnab, keda and virtual kubernetes are not azure Specific but it's actually in cncf. But yeah, you can run them all on azure as well. And just for bonus, this was actually those demos were run on Wsl2, running a custom kernel. With bx support and other stuff that i built on friday And i'm glad it held up for my demos. But thank you very much for coming out to the talk. Thank you. Thank you.


rver SX increases value. Another client reads D1 and writes back D2 also handled by SX.  Then separate clients read D2 and write back D3 and D4 on two different servers, Y and SZ. And then another client reads D3 and D4 and finds that there's a conflict, a system reports that there's a conflict to be handled. Okay, so let's practice with these. Here's two different vector clocks and you figure out whether there's a conflict, whether they represent a conflict or not. So in this case, we have server SX with the timestamp of three and on this data server SX with the timestamp of three. And then each one is a different server with different timestamps. So is there a conflict? Well, yeah, there is because on one version path, SY made a series of changes and on another version path, SZ made a series of changes and they didn't talk to each other because they don't reflect each other's changes. So yes, there is. And then this one is at the same server at a later timestamp. So is there a conflict here? Well, no, because they weren't handled by different servers. So really just this one subsumes that one and we're okay. So on this one, we have server SX with three, server XX with three, server SY with six, server SY with six, so they agree so far. And then this one has an extra change of server SZ with the timestamp of two. And so no, there's no conflict here because they agree wherever there's an extra change on top of this one. So this guy wins. Okay. And this next one, server SX with the timestamp of three, server SX with the timestamp of three, server SY with the timestamp of 10 and this has server SY with the timestamp of six and then some later change at SZ. So is there a conflict here? Well, there is because this one is later in time on SY, right, 10, than this one is. But then, and if that was all there was, if it wasn't for this guy, if it wasn't for this guy here, well, that'd be okay. We would just pick this one because it's later. But because this one's here, we now have some changes at SZ and some changes at SY that were both forward in time from the latest point that they agreed on. And so we don't know how to resolve that. And so, yes, there is a conflict. And then similarly here, SX and timestamp three and SX and timestamp three, well, here we have SY and SY, SY 10 is the same as the last one, but here instead of six, it's 20. And so it's later than this 10 and then further we have a change at SZ. And so is there a conflict here? Well, no, because this one is strictly later than that one is. On all the servers that they share, it has later timestamps. So it strictly subsumes it. And so no, there is no conflict. Okay. So those are, in the last time we talked about consistent hashing, in this segment we talked about vector clocks. These are two little gadgets to be familiar with because they come up a time and again in these no SQL systems and in other systems in general, okay. All right, so Dynamo also talks about a way to parameterize the level of consistency. And this comes up occasionally in paper, so I just wanna make sure you're exposed to it. So the idea here is that you have two parameters R and W and R is the minimum number of nodes that need to participate in a successful read, okay. And W is the minimum number of nodes that need to participate in a successful write. So this is sort of how many replicas you write to and how many replicas need to respond from a pool to know that you sort of have all the information. Right, because if everybody's updating everything all the time, it might be that you have 20 different servers, they all have a different version of the data that you're trying to read. And so the question is how many of these do you need to sample before you feel like you have the right one. And so for a replication factor of N, if R plus W is greater than N, then you can claim consistency. But often you wanna set R plus W less than N in order to achieve lower latency. So you don't wanna have to actually contact too many servers in order to satisfy some read request or write request, okay. So if you see that notation, this is what it means, but I'm not gonna describe too much more about it because I think that the sort of the formalism falls over a little bit under a little bit of scrutiny, but that's what they're talking about when you see this discussion on, say, blog posts. Okay, so the next system we're gonna look at is CouchDB, which began in 2005 but is undergoing a lot of updates and is still pretty popular today. So this is an example of one of these document-oriented stores that Rick and Tell talked about. And so here, just to look at the features we're talking about here, we've got the scale, we've got primary index. Now, we start to see the requirement for secondary indexes, meaning that you can look up not just by the key, but by other kinds of values in the document, and we'll see how that works, okay. And then transactions also are a little bit better. We can, what I mean by record here is that you can change multiple values within a single document object, which is a set of key value pairs, as opposed to just a single key value pair, okay. And then there actually is some support for analytics and we'll see how this works too. This is all through the notion, this concept of views in CouchDB, but you can actually run little map reduce scripts to compute derived new values from the existing document source. All right, and then the other notion we have here is views, which is somewhat unique. You can see this column is fairly sparsely populated, and I hit the concept of views pretty hard when we're talking about relational databases and argued that it was pretty fundamental to not only a relational model, but a pretty important concept in general, and gave you this notion of logical data independence. And so whenever you see views, that's a good thing, and so CouchDB has them as well. All right, so the data model here is documented, or as we said, where a document is a set of key value pairs. And so here in this application, these are perhaps blog posts where they have a subject, the key of subject and the value is some text string, and they have an author, and they have a data which they're posted, and then they have a set of tags, and this is sort of what makes it a document model. We said they could be sort of nested, so this is okay to have a list of objects here. And then they have a body, which is another string. And you'll notice that this is actually JSON compliant. We looked at JSON in the Twitter assignment, and so it's another occurrence of this. And one of the reasons why I wanted to make sure you were working with JSON before is it does get popular. And so CouchDB, all data is represented in JSON, and all the requests coming back for all are represented in JSON as well. So how do updates work in this context? Well, as we mentioned, you can make, it is fully transactional within a single document, so full consistency within a document, meaning that I can sort of grab hold of the document and logically sort of lock it and make whatever updates I want. Now it doesn't actually take a lock because it uses what's called optimistic concurrency control, meaning that it sort of assumes that conflicts aren't gonna happen optimistically. And so now if I check out that document and try to make updates to it anywhere I want all throughout it, and then try to commit that change, and someone else is doing this thing, has done this, has committed their own changes, in the meantime, my changes will fail. But we assume that that doesn't happen that often, so it's okay to just sort of fail in those cases. And really what you have to do in that case is check out the new changes and make whatever changes you want. Fine, but there's no multi-row transaction. What I mean by multi-row here is multi-document, right? So you can't say, for example, when I update my status, I have to update my document that describes my current state of the world, but maybe I also want to update all my friends' walls, right, their pages that reflect my new status, and you can't guarantee that that happens synchronously in CouchDB. But in this particular application and in many others, maybe that's okay. All right, you can do it in two separate steps. You update your status and then you update theirs, and there'll be a period of time when their view of your current status is out of date, but maybe that's all right. Fine. All right, so this notion of views works like this. A view specification, which is itself a CouchDB document, a set of key value pairs, but it's sort of a special one, looks like this. There's some metadata information here, and then there's this key views, which is a dictionary of things. And so this specification has three views in it, one called all, one called by last name, and one called total purchases. And each one of these views is going to be, is implementing a set of key value pairs. All right, it's gonna be implementing a dictionary. And so the all view, okay, ends up so fine. So how are these implemented? Well, speaking of recurring concepts, views are recurring from logical data dependence and relational databases, and here map reduces recurring, even outside of the context of literally hey dupe, okay. And so here the map function reduce function is actually written in JavaScript. Again, everything in CouchDB is JavaScript, and they look like this. So the all view has just a map function, no reduce function, and in JavaScript you can write these anonymous functions like in this sort of syntax. So this function doesn't have a name, it just says hey, I've got a function with no name with a single argument called doc, and the body looks like this. It says if doc.type equals customer, then emit a key value pair where the key is null and the value is doc. So we don't really care about the key in this case, we just care about the doc. All right, so this is all customers. Fine, the by last name view, you can imagine the key is gonna be last name in this case. And so here we have another anonymous function. If doc.type equals customer, then emit the last name followed by the entire document. And so now this allows clients to search efficiently by last name. And these views are in CouchDB are computed you know, they're materialized sort of eagerly and stored in these distributed B2 structure or index structures to support various lookups. And so this is how they implement those secondary indexes in that column in our table, okay. So now we can look up by last name as well as by document ID.  So total purchases here, the map function again takes in a document and says, well if doc.type equals purchase, so I'm not talking about customers anymore, just purchases, then emit the key of doc.customer and the value of doc amount. But then we're also gonna find a reduce function and the reduce function adds up all the values of the amounts. And so here we have given a customer, if I give you a customer, you can return it's the total amount of all of its purchases, okay. And CouchDB maintains all these views as things change. With eventual consistency, sort of semantics. And so we have a lot of things going on here with one concept. We have secondary indexes, we have logical data independence. We have map reduce computation. Okay, so I had checked the box in the table saying that CouchDB could do joins in analytics and it's not really quite true, they're somewhat limited. So let me show you an example of how they do sort of joins. They have this concept of view collation and what you can do here is write a map function in a view that looks like this. And so here we're trying to group together all the comments associated with a post. So it's sort of a join between the comments table if you will and the posts and the blog posts table. And so this map function is the same kind of thing we did in the map reduce assignment to do a join. This is what you have to do to join a map reduce is you pretend like the whole collection of documents regardless of type, regardless of source relation, here's one big set of objects. And then your map function, you sort out which one's which and make sure to hash it on the same key. So here the ID of the document post and then in this document doc.post refers to some post ID. It's now post ID equal to one and all the comments associated with post ID equals one will go together. But they do this funny thing. They say the key here in the key value pair is document ID followed by the number zero in the one case and the number one in the other case. And what's happened here is that all everything in CouchDB is sorted. And so you've got things sorted by document ID and then sorted by this extra bit. And so now you have the post ID coming first and then all the comments coming second. And now what you could, you know you think you could just do a reduce function to actually compute the join that you might want but it turns out reduce functions don't quite work that way and there's some scalability issues. So what people recommend is to do this view correlation trick where you still spit out the data in sorted order but then you can query it. Say a client application could query it like this where they say start key is just the post ID without the extra prefix at all and the end key is post ID with the number two. And so now this gets the full range of the post ID which has the key of zero followed by all the comments with the post ID of one. So you got everything in one big list and now you could process it in order to show it, say showing the application. So you're essentially constructed the join by cheating with the order. Okay, so fine. So perhaps you could argue that you have some joins in analytics but it's a big glib to say so. Okay, so the third influential system that Rick mentioned in the paper is Bigtable from Google which is a paper from 2006. And so here we're looking at primary index lookup, secondary index lookup, transactions are also at the sort of scale of an individual record. Joins analytics is not supported by Bigtable directly but in HBase which is the open source implementation of it and in Google's implementation as well it was sort of designed to be compatible with MapReduce. So you can run MapReduce over here over the same data that's stored in Bigtable. So there's sort of complimentary. And then there's some notion of integrity constraints or schema here and we'll talk a little bit about how that's implemented. There's no views that I can see and there's no sort of language level or algebra level for manipulating these things or all sort of no SQL style micro interactions with individual records and cells. Okay, so this is a paper in OSDI from 2006 and some overlap with the authors of the MapReduce paper and it was sort of designed from the start to be complimentary of MapReduce. So if you can remember what was one of the main things that was missing from MapReduce or maybe a few things that were missing. Well, in particular you couldn't look things up by index, you couldn't get these sort of low latency accesses. And so for example, if you wanted to find all the records, given a big data set, you want to find all the records and some other data set that correspond to it. You want to do some sort of a join. The best you could do is process, you had to touch every single record. There was no way to zoom in to just the right ones you wanted. Okay, and so Bigtable provides that fast key-based lookup but you can still process the overall data as a big set of key value records with MapReduce. Fine. So the data model here is a sparse distributed persistent multi-dimensional sorted map. And what they mean here is that you can basically access any cell in a Bigtable by giving a row ID, a column name and a time stamp. Time stamp isn't really described in this English description here. It's for versioning. So after you have updates you'll keep track of past versions of the same cell. Okay, and so if you provide these three parameters, Bigtable will return you a string quickly. So each row is, data is all sorted lexicographically by the row key, which is this row ID in this bit, right? So this is some sort of primary key in sort of relational language or just a key in kind of no SQL framework. And then this key range, I say the integers, and contiguous sub ranges of this set of keys will be assigned to a tablet. Okay, so this is a little different way of dividing up the data than we've seen in the past in at least one system. We talked about a parallel database model. We happen to use the example from Teradata. And so how did they break up data? Well, they did it by hashing, right? So every individual record would be sent to a server according to a hash function, which you can generally just think of as sort of round robin. The point is is that two keys that are next to each other in space, so sort of timestamp 5 p.m. and timestamp 501, there's no reason to believe that five o'clock and 501 are gonna be on the same server in Teradata's model. Here they are. So what are the pros and cons with this? Well, if you're going to typically access a whole range of keys at once, it's pretty nice to be able to, when you get one, you get the others two sort of for free because you're pulling them all back. However, if one particular key range is much more popular than the others, but using the time example again, the most recent data perhaps is the most popular. And so if all the requests are going to that one key range, then you've got a bunch of idle servers hosting all the other tablets that are corresponding to older times and all the requests are going to this one tablet. And so for that reason, Teradata sort of chooses to hash everything so that on every request, all the servers may have to be accessed, but that's good for scalability. Okay, so pros and cons. All right, so the tablet here is the unit of distribution and load balancing. So they'll move tablets between servers as things start to get unbalanced. If your key range is January, February, March, April, May, and there's a whole lot of data coming in from March, they'll split that into multiple tablets and start moving those tablets around between servers in order to balance things. Okay, so within a single table, you can have these groups of columns called column families and the column names have the family right in there as a qualifier. And this family is the basic unit of access control so you can provide permissions on a group of columns, memory accounting, and that they're sort of allocated as a unit in memory and then disk accounting. So they're moved around on disk as a unit as well. Okay, and so I think this point that typically all columns in a family are the same type, which I find a little unusual because they sort of talk about it being the basic unit of access control is to suggest that there's, you know, things that go together for access control, sort of social security number and employee ID or something may or may not be the same type. So they're sort of a logical grouping requirement that they seem to be trying to meet, but then they're the same type, they have to be the same type which is for a very technical reason, specifically because they wanna compress these things. So if you have a whole bunch of integers, it's easier to compress than if you have a mix of integers and strings. Okay, so I think they're trying to kill too many birds with one stone here. All right, and then each cell can be version which is the third part of that key lookup, right? Row ID, column, name, and timestamp. And each new version increments that timestamp. And so here you can enact different kinds of policies where you only keep the latest in versions or you keep only the versions since a given timestamp. So how these tablets are managed is a master will assign the tablets to tablet servers and the tablet server handles region rights from the tablets it controls, okay? And so clients communicate directly with the tablet server as opposed to having to go through the master every time which helps with scalability, okay? And then when a tablet starts to get too big, it'll split it and load balance it. All right, so the metadata keeping track of where tablets are located is organized itself in another tablet. So there's a root tablet here that describes a, each record in here describes a group of records, a group of location records in a, you know, bigger table. And then each one of these metadata tablets gives a location of a particular user table, okay? And so those values sort of keep track hierarchically of where everything is at one time. And so Chubby that they mentioned in the paper is a distributed lock service for controlling access to these things. I'm not gonna talk too much about it. Okay, so how are region rights handled in this system? Well, there's a table in memory that stores a sequence of updates as they occur, okay? And a write operation is, you know, adds a record into the memory resident table but it's also written to a log for fault tolerance purposes. Okay, so if this goes down, it reads a tablet login can reconstruct what's going on. And then read operations are served by reading these SS table files, you know, the actual data itself, but then also by applying the updates from the mem table on the fly. Right, so it needs a stream, it says here's the value and then here's the stream of updates I need to apply that value to get the true value. And then there's two, so this is fine, but what happens when the mem table gets bigger and bigger and bigger? Well, there's two types of events that occur to do the bookkeeping here. So one is a minor compaction and this is when the mem table gets big, it gets written out into a new SS table file and the changes are merged. Okay, and then a major compaction is take all the SS tables and rewrite them all into one big one that may be split into multiple files and also clean up any deletes that have occurred. So deletes are just appended as instructions, but aren't necessarily, it doesn't actually remove anything.  So this way you can keep sort of the read throughput pretty high and sort of keep this upkeep going on in the background. All right, so there's a host of other tricks here too where they can do various forms of compression specified by compliance, which can be specified by the clients or some different ways of doing it. They use Bloom filters to speed up existence tests. So if I give you a row ID, a column ID, and a timestamp and say find me this value, what these Bloom filters allow you to do is to very quickly determine whether that does not exist in the system. So these Bloom filter data structures are pretty cool and I'm gonna walk through them in this course in a couple of weeks, okay. So they help you quickly determine whether that key does not exist in the system and avoids disk accesses during reads, all right. And then there's locality groups you can define which another layer of organization on top of families. And these are groups of column families that tend to be accessed together, fine. And then another trick here is to make sure that the SS tables, these disk chunks are actually immutable. They never get written directly. The only time they get written is when these major compactions happen and the whole thing is sort of reorganized. And so that means that the only writable data structure is this MIM table. And so the amount of concurrency control to keep things down to the remains pretty simple. Okay, okay, so Google Bigtable again had a lot of influence and one of the major outcomes is just like MapReduce it was taken up and implemented as an open source project called HBase. And so where Bigtable was compatible with MapReduce HBase was compatible with Hadoop. And in one slide there's not much difference here but I just want to mention the terminology so you've seen it. That there's table at the top level and then region, store, and then MIM store and store file. And so the exact names of these structures are a little bit different and they did insert one more layer of abstraction which is this region, fine. And then how this sort of is compatible with MapReduce is that each one of your map functions will process a single tablet. And so it's sort of one to one with the blocks of data that we talked about when we talked about MapReduce. And then I just sort of ask this question sort of open ended is that there's this notion of speculative execution of MapReduce that we talked about whereby for fault tolerance reasons it might kick off the same map task twice on two different replicas of the data. And the reason for this is that if one fails, well, you have the other one, right? So you don't have to start over from scratch. But in this environment where you're now working on data that is being actively updated and that's what HBase and Bigtable are designed to support is updates, it's not quite as clear to me what's gonna happen when you have, it's possible and because of eventual consistency that these two tablets will not always agree instantly on the same record they'll agree but different records within the tablet may not. And so just an example of when you sort of mix these two systems and there are no sort of system-wide transactional guarantees or system-wide even properties that you can run into trouble. And I think that a general theme here with these sort of no-SQL systems including the ones implemented or designed by Google is that you're kind of offloading some of those responsibility to the application, to the programmer to sort of sort of sort this out and make sure it's okay. Okay, and we're gonna come back to that in just a couple of minutes. All right. So after Bigtable, several years later, there's a paper by a bunch of folks at Google about a system called Megastore that I'm not gonna spend a lot of time on but it's basically, you know, they found this point I sort of just made is that these loose consistency models can complicate application programming and what they wanted to do is provide a little more system support for certain kinds of safe updates. Okay, so here instead of full transactions being safe within an individual record as they are in Bigtable, they've extended it with this notion of entity groups. And so an entity group, this should be on this slide, and an entity group is a set of records that tend to go together, tend to be accessed together. Okay. So maybe again, this is the blog in all of its comments, for example. You each one of these is a record, is it's like sensible record store, so it's okay for them to have different schemas but they all tend to go together. Okay, and so what they've done is extend transaction support over an entire entity group, you know, a set of related records. Okay, so they still get the scalability by not requiring full system wide to global synchrony, but they allow you to sort of, they get away from this problem of, you know, very frequently I might need to update one record and then update all of its sort of children records at the same time, and I can't do that in any kind of safe way. Okay. So fine. Fast forward one more year. And so there's a 2012 paper on a system called Spanner, and I just want to mention these quotes and then we'll talk a little bit about the system and this one is still sort of being explored by the community. It's not available actually for use but the paper is being explored and the idea is being explored. So for example, you don't see an open source, actually that's not true, you do see, there has been a couple open source and limitations of the ideas in Spanner, but they're not quite as popular as some of the open source and limitations of the other Google systems. Okay, so, you know, it says even though many projects happily use Bigtable, we've also consistently received complaints from users of Bigtable, it can be difficult to use for certain kinds of applications. Those that have complex evolving schemas are those that want strong consistency in the presence of wide area replication. Okay. And so then they say, go on and say, we believe it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise rather than always coding around the lack of transactions. And so this, you know, the database community could have said, you know, well sure, you know, well duh. Right, that's exactly the point. Is that system supplied support for transactions is always a win, right? Because it's difficult, error prone and expensive to try to do this at the application level. And more importantly, it's fundamentally wrong in some sense to do it at the application level because the application doesn't have global knowledge of what's going on. Right, only the system does. So fine. So all those Spanners scalable and the number of nodes, the final quote here. The node local data structures have relatively poor performance on complex SQL queries because they were designed for simple key value accesses. And then algorithms and data structures from the database literature could improve single node performance a great deal. Again, you know, it's somewhat of a Google style approach to the problem of reboot everything, rebuild it all from scratch and then sort of cherry pick and bring things in. So this has been working pretty well and they have fantastic impact in the community. But there's a lot out there in the database literature and in the database systems that could have been used from the start. I mean, in fact, trying to start from the beginning and just say, we're going to build a big Google style parallel database may have been a good choice rather than sort of getting completely away from it and then coming back incrementally and find yourself in a SQL system. Now I sort of skipped over what Spanner is, but it's planet scale database system. There is a SQL like language.  I'm missing our table here. Let me flip back a few. So here it is down here. So I'm missing the slide here where I showed it. So really big scale. Primary accesses, you can access by other attributes. There are transactions and in fact, they're global this time. They're real asset transactions. It's not clear to me whether joins are supported. I suspect they are because I keep talking about SQL but I couldn't find an example of whether it is or not. There is some notion of schema and they do sort of protect against data that doesn't conform to the schema. There is some notion of logical data independence, although they don't talk about it much. There is a SQL like decorative language on top of it. I didn't see much evidence that they're doing a whole lot of fancy optimization and I did just show you that quote of where they say that their performance is sort of poor on complex analytic queries, but that's something that probably could come along somewhat quickly. Okay. So fine. Spain are at a high level. Let me give you a couple more details about what this system does. So the data model here is this notion of directories and these are a set of continuous keys with a shared prefix. So you can think of kind of like a tablet was in big table, but now they have this notion of multiple logical tables being sort of interleaved. And so if you're not used to staring at this syntax don't worry too much, but those of you who are thinking in terms of DDL in a relational database, they have kind of a create table language that looks like this. You create table users with two columns and then you give it this keyword directory and then you have a create table albums with some columns and you have this keyword interleaved in parent users. And what you end up with is something like this where there's a user with all of its albums and a user with all of its albums. And so you can see here that, you know what we've been talking about all of these different systems are experimenting with ways of getting these nested data structures. Hierarchical data structures that look a lot like what we saw way back in the 60s, right? And the motivation is the same as it was then. It's actually really, really fast. When you're gonna access, when you wanna pull up a user and then immediately pull up all of its albums it's really fast to access those this way, right? You know, but I'd probably speculate that the reasons why relational, the relational approach, eventually replaced these, it can and will happen here as well is that performance is not the number one priority a lot of time. It's minimizing the amount of developer headaches. So, it may need to be seen, but I think that this incremental walk step towards a big new scalable relational database is underway. Now again, that doesn't mean that I'm saying use all the old databases. They really were designed for sort of a different workload and they really don't, there really is no evidence that they scale to some of these levels. But that doesn't mean that you sort of throw out all the, everything we learned, okay? But that's more me editorializing. So fine, so how this works is there's a universe master at the very, very top and this is just a singleton. There's only one of these for deployments and they sort of imagine there might be only one or two of these deployments anywhere. So they have sort of one for test, one for production slash test and one for production and that's it. So many different applications will use the same deployment of Spanner. And so this is mostly just status information about the zones. It doesn't interact with clients at all. Then there's a placement driver that's responsible for moving these directory sets of records around for load balancing purposes and this happens on the scale of every few minutes. All right, and then within a zone, there's a zone master that assigns data to Spam servers and then there's a location proxy that sort of knows where everything is and routes requests to the appropriate Spam server and then the Spam servers themselves serve data and so in here it's starting to look a little bit more like big table. You know, zone is essentially an individual big table deployment, okay? So inside of a Spam server, this is where the big difference here is this is where they're gonna try to support fully consistent transactions. So across these, you know, within a group of these replicas, they support two phase commit. This only is needed when a transaction actually accesses data that's across is not constrained in one particular replica, okay? Other than that, it just skips over this logic and it doesn't cost anything, okay? And then one step down below this across, so this is, sorry, I guess I'm using the word terminology. So two phase commit is across groups and when all of them, the transaction only is contained in one single group, then you drop down a level and you run the Paxos algorithm that I didn't talk about in detail but I mentioned exists in order to sort out the reads and writes in order to handle the write, okay? Okay, and the only thing, the other piece I'll mention here is that this term Colossus is new, it's the successor to Google file system and Google file system is the original term for the open source implementation of HDFS which underlies MapReduce and Hadoop, sorry. GFS is to MapReduce as HDFS is to Hadoop. So when I'm throwing these acronyms, actually, that's how to keep it straight, okay? So that's all I wanna say about Spanner in particular, we just take a step back and look at all the different systems that Google has for a second, you know, MapReduce was a paper in 2004 that had a ton of impact, Bigtable had a ton of impact and then later there's Megastore, there's the system tensing that we didn't talk about but it's a SQL system on top of MapReduce much like Hive if you were familiar with that or if you remember we mentioned it and then Spanner very recently. And so you can sort of organize these things into a timeline this way just to kind of get a sense of this and because of these systems that have so much influence I want you to be aware of what they are and sort of how they fit together so it doesn't just sound like a big jumble of terms. So MapReduce was, you know, one of the earliest ones wasn't quite the same, there's actually another one called Sawzall that didn't really get a ton of traction but it was a nice paper. And then Bigtable came a couple years later and I drew a dotted line there representing that they're sort of compatible or designed to go together. You know, one was for MapReduce for analytics, Bigtable is for the sort of micro operations. And then both of these a few years later had an open source implementation in Hadoop and HBase respectively. Okay. Fast forward a few more years and you've got Megastore and Spanner coming very quickly one right after the other and this heavy blue line represents, you know, it's pretty clear that the influence is fairly direct. In fact, I would suspect that there's a lot of code being borrowed and then Megastore makes plenty of references to Bigtable and Spanner makes references to both Megastore and Bigtable and the papers have many, many of the same co-authors. Okay. And then MapReduce depends directly on, sorry, excuse me, Tenzing which depends directly on MapReduce. It provides a SQL layer on top of MapReduce. And then there's some other systems here. One is called Dremel which was originally for very fast aggregate queries but really just aggregate queries but at extremely low latency. So this is, you know, in the analytics camp because you're doing these sort of aggregate questions opposed to sort of micro updates but it was extremely low latency unlike MapReduce which is more of a batch system. And so this is a great fit and it's a very nice system. And in fact, since then they can do joins not just aggregates and more importantly this was exposed as a service that you can just use directly over the web even in your browser called BigQuery. And that's an important one to watch. It's one of the few systems that is available as a service through a, you know, as a cloud service but scales to very, very large data and sports analytics. Okay. And then another one that we won't talk about yet but we will come back to is Pregel. And this adds the one secret ingredient that is sort of near and dear to my heart which is iteration.  But for many kinds of tasks especially in data science many of these analytics tasks these machine learning tasks you have to do something again and again and again and again until some kind of convergence condition is reached. And Pregel and one of our systems and a few other systems are the ones that try to notice this limitation of MapReduce and Extended. So people are doing this with MapReduce but they do it sort of in fairly ad hoc ways. Okay. And so we'll come back to that and talk about it but, you know, analytics, low latency, micro updates so there's two big classes of systems and then analytics with iteration is perhaps a third class of system that we'll talk about. Okay, so going back to the grid we can highlight those systems that were based on MapReduce itself. You know, the MapReduce paper itself in 2004 and these language layers on top Pig and Hive in 2008 where Hive is SQL and Pig is a relational algebra looking language that we'll talk about in some detail in the next few segments and Tenzing, which is also SQL and Impala, which is also SQL where Tenzing is from Google and Impala is from a company called Cloudera that's a pretty eager evangelist of MapReduce and Hadoop based technologies in general. Okay, so one trend I think you see is that these declarative languages on top of the parallel processing primitive of MapReduce are really here to stay, right? So people that were relatively against these kind of languages are certainly doing it. Now, it's also fair to say that the enterprises in general have made a pretty significant investment in SQL expertise. So even if they're attracted to the advantages that Hadoop might bring, they're pretty much demanding SQL. So this may be a response to this inertia from having invested in SQL in the past. I think that's certainly true. However, it's also true that the desire for declarative languages is reasonably well founded for reasons that we've already talked about. Okay, so you can put these systems on this timeline and the only point I wanna make about this is that there's a bit of a gap between the paper in 2004 and these systems in 2008. But then as soon as you had Hadoop, the system itself developed a Yahoo and released as an Apache open source project, you sort of immediately see an ecosystem start to emerge of extensions to it that add these, in particular adding these languages on top. And so I think that the need for a high level interface is motivated by how quickly they came around as soon as Hadoop was out. And again, it didn't stop with these later systems a few years later. Okay, and actually not on this page, there's potentially hundreds of, if you include research projects based on extensions to MapReduce, there are really, really a lot. Okay, and so this is some of the most popular ones. All right, so another subset of this grid that you can look at is just the NoSQL systems. Now, the whole last few segments have sensibly been about NoSQL, but I've also included these kind of analytics systems in here, MapReduce based systems and a few others, for example, Dremel and Spark and Shark. Dremel is a system from Google that is the back end of a query as a service system called Google BigQuery, which is pretty nice. I'd recommend taking a look at it. You can sort of upload data and put it in there and it doesn't matter how big it is and you can kind of query it at very low latency speed. Spark and Shark come from the AMP Lab at Berkeley and are part of the Berkeley Data Analytics Stack, BDAS or Badass. And Spark is a language level on top of, it's not MapReduce, but on top of a parallel processing system. And Shark is an SQL layer even on top of that. Okay. And so a couple of the distinguishing features of Spark is it loads everything into memory, processes everything there when possible, writing things out to disk only for fault tolerance reasons, so much less often than MapReduce. And it also supports iterative processing, which is pretty important and we're gonna come back to that a little later in the course. And then Shark again is just SQL on top of this. All right. So within these NoSQL systems, one thing you can look at, excuse me, there's not much I wanna say about this diagram, except that to point out that there's been sort of the cambering explosion. So first you had sort of memcache, which is again just this caching layer for really the real system. And the real system was a bunch of my SQL databases that weren't really working all that well for the requirements they're having used for, but you could bring things into memory and keep it there, looking it up by name. And it was just sort of a performance enhancement, a free performance enhancement if you invested in this system. The real approach of throwing out everything you have and replacing it with a NoSQL system came a little later. And so the only couple of points I wanna make is that there's been kind of a cambering explosion of different systems around this time. And that this space in here is not as nowhere near as empty as it looks. I picked out a few systems here, but really the emergence of new systems in the space hasn't really slowed down much at all.


Welcome once again to the Rugged Rockhound. Today we've got a bunch of rocks to cut open. Let's get at it Today we're starting off with some of the black agate. Had a nice thick vein on it That's pretty much all there is. Well, okay, there's little veins through it, but mostly it's just that big thick vein Little bit of Bertrandite Okay Hey, you got a little bit going on in there. That's pretty This is another one of those really nice malachite or copper ores from I think it was Eureka Unfortunately, this side just shattered. So we'll just take a look at this side then Okay, I mean the inside looks nice, but I would say the outside actually looks better. It's more green Here's this nice piece of agate that my subscriber sent me last year. It looks really cool from the outside Mostly this side that red. So here we go cut open Where'd the red go? Okay It's interesting Looks like the red wants to stay on the outside though. I mean it's still pretty Here's another agate I had lying around Yep, this one's an agate. You can see the nice banding in there Not a lot of color, but cool nonetheless Another flat dugway geode Looks good pretty much par for the course. Here's some more black agate. Oh There's a lot in there Now this should look really nice once I grind this side down to really kind of let the light through That should end up being really nice It's more Vernon Wonderstone Still pretty So this is some more birch rindite, but without the purple and it broke when I was cutting it So here's just a piece and here's the main part that survived Still pretty like even without the purple the patterns are nice Here we've got a nice big piece of Jasper Let's take a look Nice got a lot of cool little dendrites in there. It is very dark though There's another good-looking agate that I had just sitting around Okay It's got a little bit of color in there a lot of quartz right in the center of it that is growing Here's another piece of the birch rindite. Oh That's nice. I like that one. That is a good one There's another flat geode you can probably hear the jet flying over Oh nice Not too bad Here's a long skinny piece of the Vernon Wonderstone that part broke off And that's what we have Nice and last one for the day was this big beautiful piece of wood that as I started to cut it It said nope break So I'll just show you the broken pieces They're very pretty though Still very pretty Hope you enjoyed that there were some neat stuff in there It was and it's just always fun to cut open rocks and of course until next time remember there is treasure everywhere


Live from Seattle, Washington, it's theCUBE on the ground, covering KubeCon 2016, brought to you by the Linux Foundation and Red Hat. Here's your host, John Furrier. Hello and welcome to theCUBE on the ground here in Seattle for a special presentation of KubeCon and CloudNativeCon. This is theCUBE's coverage. Our next guests are Josh Burkus with Red Hat and Perdipto Pachaharia, who's also with Red Hat. Guys, welcome to theCUBE, KubeCon and CloudNative. I mean, there's now events going on around this hot stuff. Thoughts on this. Obviously developer craze is all where it's all about.  It's truly changing the game right now and the developers are driving all the action. What's the conversations like? Obviously orchestration, Kubernetes has more job openings and I think then I've seen in a long time it's like straight up. There's so much action in Kubernetes and containers. Well, it's as, see, I've actually been on the Ops side. I've been in Ops software geek for 20 years. This is actually very exciting for me because if you had told me even like six or seven years ago that we'd get, what do we got here? 2000 people at a conference to talk about automation and distributed systems. I'd have laughed at you. Be like, oh, you get like 100 guys at Berkeley to talk about that. Not like large groups of developers. So it's very exciting that we're actually like, trying to move forward the state of the art of software. Not just what we do, but how we do things. Pradipto, you know, one of the things we were talking about earlier and all in the queue all the time is we've been all big fans of DevOps going back since we started SiliconANGLE. But DevOps now is so mainstream and actually have a name for it. It's called Cloud Native. So Cloud Native has kind of become the DevOps word for just how things get done in software. Right, I think it's the generic software thing that you come up with new term to take it forward. But it's just amazing how people started from Agile then they go to Cloud Native. They went to the DevOps whole revolution and now they call it Cloud Native. It was amazing that they just announced the next conference right now, which is a year from now. And it will be apparently two times bigger than this or three times bigger than this. And they already announced the Berlin one. So it's obviously it's like global thing across everybody, across everywhere where we need so many people and so many people are obviously attending the conference. It's like full house there. I had to take a whole panorama picture of all the people in the whole room. I mean, to me, it's so exciting for me that I was one being a software guy and seeing this kind of celebration of the tech. So it kind of makes me kind of get a tear in my eye. But more importantly, as an entrepreneur, there's huge wealth creation and value creation around software right now.  And that's a sea change. Now, that's all great. I guess everyone excited, but you got to operationalize it. This is where the challenges are. So, Josh, talk about the things you're working on. Project Atomic is one. This is a big deal, this operationalizing, this cloud layer thoughts. So, and a lot of it is, we're looking at how do you actually do en masse things that people didn't at Boutique way before, right? 10 years ago, you wanted to actually scale out a system and make it automated and make it a real distributed system.  So, one of the things we're working on for Project Atomic is when you start looking at these container-based orchestrated systems, what you need out of the operating system changes, right? Because you're not installing individual operating systems and then configuring them by hand anymore. And configuration management kind of brought us towards that but not really where we want to be. So now we're looking at, hey, if I'm looking at an individual host, not as an individual host, but as part of a whole group, how do I want to manage that? And that's what Atomic is about, is that we want to actually manage the individual hosts en masse the same way that we manage the containers. That's Atomic host. That's Atomic host, yeah. And what's your goals for this year? I mean, obviously it's a lot of community rallying and going on, what's your goals? So, our goals is to actually, well, two things. One is to add the rest of the features that we need on the server. And we just recently made a milestone in that so for both the RHEL Atomic host 7.3 and for Centos Atomic host and Fedora, the open source versions, we added layering so that you can actually deploy because Atomic host, it's an imaged OS that is the OS gets installed from an upstream image from a server so that you can distribute en masse rather than configuring on site, right? But you need to make changes to groups of servers. And so we added a layering feature so that you can layer extra things on top of that image in the same way that you would inside a container, but this time in the host OS so that you can actually have groups of servers that have special purposes because they need extra software like storage software or one of the big things I've been using it for is to install different versions of Kubernetes because I also mess around a lot with Kubernetes, right? And so I want to have 1.2 in the server and 1.4 in the server and 1.5 alpha in this other server. So it allows you to focus in on things you want to get done. And how is this reinventing the OS because this is a shift in thinking. Right, well, the thing is that basically the way that we have distributed managed software for the operating system really hasn't changed in any fundamental way in almost 20 years, right? It's been packages, right? RPMs or Debs or whatever, getting distributed to stuff, getting installed and then getting configured, right? And we added some configuration management here, but we didn't really change how we actually do things. And so moving to an OS which is based on an image tree really changes that. And also with the idea that the base OS should be lean and that the stuff you're installing on top of it should be installed in namespaces, containers and on the desktop, there's a project called Flatpak from the GNOME for installing desktop stuff. So one of the other things that's coming out of this is because this came out of Red Hat because Atomic came out of Red Hat, it's all been very focused on the server. But in the meantime, a lot of our users in the open source space particularly in the Fedora project are like, well, I want to use this for desktop because like I want to run the latest cutting edge version of Fedora but I want to be able to roll back and recover if it stops working. Yeah, what about security too? This is another concern, security. Yeah. Yeah, and that comes from sort of a different place. I mean, obviously big enterprises are now looking at automating in this way. And a lot of them had actually gotten very used to the level of isolation that virtual machine based infrastructure gave them or real machine based infrastructure gave them. And containers don't deliver the same level of isolation. And so we need to rely a lot more on security tools built into the OS and built into our container stream. That doesn't disrupt Project Atomic. It facilitates it, doesn't it? It facilitates it, yes. Yeah, because we're looking at building in all of these sort of tools, right? And so Project Atomic is sort of an umbrella and like one of the tools we have under there is Bubble Wrap, which is a way to actually run non-root containers even under Docker which has some issues with running non-root containers. So that you can limit people's permissions. We do a lot of work with SE Linux because the Red Hat platform is devoted to SE Linux in order to do that. And lately we've been doing a lot of work on image identity and image fingerprinting to know that you're getting the image that you think that you're getting and that it's been verified. And that's a big deal actually in the whole container world but it's very important, right? We have a tool chain for RPMs and Debs and regular packages for that. We need a new tool chain for containers. So real evolution and tool chain. Pradeep, to talk about open compose and what you're working on specifically at what stage is this project? What's the update? Sure. It's a very new thing. It was literally released like the initial cut or the POC version was released like last week. So the idea is Kubernetes is awesome and but it needs sometimes you have to do a lot of work to get it. So a higher level abstraction is always a nice thing to have at many times which will help developers get to use Kubernetes or deploy their applications on Kubernetes or the microservices very easily. As a result, we thought about it how it can be done. Can we create something abstraction that can be fed into tooling maybe and then deploy the applications? That's how the idea came into picture first of all. So what we have done is we have created a back for now and a tool and we have been experimenting with it internally with some various microservices that we are building internally and we're trying to deploy them on Kubernetes or OpenShift and it works great for now but we would like other people from the Kubernetes ecosystem and other orchestrator maybe even if it can be made. Composability is what everyone wants. They want to have it basically easily to compose apps and who's involved in this right now? So you know Red Hat's behind it. What causes it? We are talking to Google already. We are talking to, we have our initial conversations with CoreOS as well and then there is a smaller company called as Skipbox.  We would ideally love to talk to MISOS and the MISOS ecosystem as well if it makes sense but the whole idea is as you said composability of services and tooling around. What's the main attraction to get people on board with this? High level abstraction, make it easier. The user experience should be really, really simple and easy to use and it should be very simple to teach anybody to get on board on this. And where's some information that can get GitHub and code out there? Yes. Give us a quick word on that. So there is a repository called as Open Compose inside the Red Hat Developer Organization. That's where it resides. Now the spec resides there. The tooling and basically the read me should tell you everything. It has a link to the toolings and everything as well. Plus, we have written a detailed blog post including some code examples on the, we can share that link with you. So that actually can take you to the details of the project, yes. Josh, final question. How does Atomic, OpenShift and Kubernetes look like? How does it all come together? Okay. So Atomic is the underlying layer for whatever you're going to run as your orchestration system and your application management system on top of it. The idea is to actually provide a really solid base for that. So like when Red Hat, if you're doing OpenShift online and you're getting supported OpenShift from Red Hat, it's running on top of Red Hat Atomic. But say if you don't want to use OpenShift, you have a different tool chain, you want to use Kubernetes, you know, mainstream Kubernetes for that without the addition of the OpenShift tools, then we're going to actually run that. And like one of the reasons for adding this new layering feature was that you could run whatever version of Kubernetes you wanted. I've got a microcluster down in the booth running 1.5 alpha right now. Guys, thanks so much, OpenCompose, Atomic. Kubernetes, just quick, give you guys the last word. How hot is this area? I mean, give some color, personal opinion, pontificate, opine, share your thoughts on what's going on. It's big. I mean, like I said, in the future, everybody doing just about anything is going to be doing a distributed system and Kubernetes gives them the tools that regular ops people, regular devs can actually do distributed systems and have them work. I would call it democratizing the cloud, I think. It makes it easier. You don't need to be the sysops guy who knows everything. It makes it like... It's an operating systems guy's dream. That's supposed to my degree and undergraduate. And I love the cloud as an operating system. If you think of it that way, the world is your oyster, as we say in Seattle. Thanks so much for coming to Kubernetes. We're in Seattle. We have the best oysters and the cloud is one big operating system. I'm John Furrier, you're watching theCUBE on the ground. Thanks for watching.