Okay, we are going to get started. Hello everyone, thank you so much for joining us for today's CNCF webinar, Managing Your Policies and Standards. I'm Jerry Fallon and I will be moderating today's webinar. We would like to welcome our presenter today. I'm Ed Badrun, Chief Technology Officer at Megalix. Just a few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee. There's a Q&A box at the bottom of your screen, so please feel free to drop your questions in there and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such, just subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of the Code of Conduct. Please be respectful of your fellow participants and presenters and please also note that the recording and slides will be posted later today to the CNCF webinar page at cncf.io slash webinars. And with that, I'll hand it over to Ahmed for today's presentation. Thank you very much. So we'll get started here. Thanks everyone for joining. I know it's a bit early for some of you. So today we'll be talking about managing policies and standards. We'll dig deep into that to see what that means. But mainly really, we're gonna be talking about two key technologies here that are open source part of the CNCF, which is Regal OPA, as well as Gatekeeper. So just a little bit about me before we get started and talk about this topic today. So my name is Ahmed Badran. I'm the CTO at Magalix, which is a startup that's about three years old now, specializes in kind of this area of governance and operational excellence for people joining, kind of going through the journey for cloud native. Prior to that, I did a while, some time ago at Amazon AWS, back in the day when all of AWS team used to fit actually in one floor in one building. Now they occupy almost a big chunk of Seattle. Went between different companies and it was interesting going through kind of the journey myself, kind of this cloud native journey even before the term existed, which is kind of what many of people now going through as they migrate kind of their legacy infrastructures and system and the monolith into this new world. So let's jump into that and kind of talk about really some of those cloud native challenges, kind of to create the context and background before we start talking about policies and standards and why should we even care. So when you think about kind of the old world of especially monolith where you have this single kind of big monolithic app where all your requests come in. So operationally, it has some interesting characteristics to it. It's simple to deploy, it's one thing, but also it's a little bit of a challenge because you get a lot of different components of your system in the same application, one part fails, everything kind of fails. It's either one or all. And then as people start migrating kind of to the microservices architecture and then sort of kind of adopting the dev ops kind of methodologies with these changes and these kind of journey that people are going through come also some challenges. One of the key things here that people maybe sometimes don't realize until they're really into the full implementation and productionization of their new microservices architecture is as you distribute your architecture and divide the responsibilities, you also dividing and distributing your problems. You're trying basically to decentralize the decision-making in your system. And that comes with its own little bit of unease, especially in the operational side because I used to have one thing that I know how to deploy. Now I have so many different things that I have to kind of make sure I'm aligned with the dev teams and how these different things should be deployed. There is now many different ways of people configuring their services, setting up things and definitely the containerization evolution, revolution if you may, helped with that standardization a little bit but also different teams have different ways of doing their thing, their setup, their configuration. So it creates a bit of that tension between the developers and the operator because at the end of the day, the reason we moved to the microservices, the whole cloud native promise is that agility. We wanna move fast. We wanna be able to iterate on a lot of innovative idea as a business. And that's why you separate kind of, you decentralize your decision-making and you let kind of your development team kind of be creative, innovative and move without having to kind of synchronize everything in a single monolith or a single, even organizationally you become agile and you become distributed. And that's kind of what the development team wanna kind of push forward with.  Now, there comes this tension where engineers wanna move fast but then operators and the operation team wanna make sure things follow certain rules and standards. So it's not a zoo. If you wanna, everyone need to play a good citizen of this distributed infrastructure. We all share those resources. You know, one application running away with their memory or their CPU just impacting somebody else is not a good thing. So there comes this tension and it's obviously not a good thing to have your devs and ops kind of not in the same page. That's the opposite of the dev op kind of, you know, culture you wanna embed. You wanna people to be working together and synchronizing or kind of synergizing the effort to really solve all the business problems as opposed to be fighting. You don't wanna your operation team to be the bottleneck trying to review every change that goes out to make sure, you know, everybody's following whatever the standard is. So where does these standards or policies come from? Well, this could be best practices established by your team. They could be tribal knowledge you've built over time of how you wanna name maybe having a name and convention for your services or what have you or it could be security related configuration and best practices you wanna follow or depending on what industry you are in that could be compliance. So regulatory legal things you must kind of, you know, check the boxes for. Again, it's not, it's not a, you know, a problem that is only a few people faces. It's everybody probably going through the journey that they go through that and they struggle through that. Now, this is just one of those challenges. I'm not trying to kind of say this is the only challenge people, you know move into the cloud native, you know, paradigm is facing. But this is certainly one interesting one that tend to slow down and kind of, you know, kind of read its head a bit as people kind of doing migration and people kind of hit it a little bit late in the game sometime if they haven't thought about it because you're trying to move fast. You just microservices, you know you need all the technical things and it's like, you know this is utopia at the end of the tunnel but you end up facing some of these kind of really practical challenges that's going to hit you that affect your culture and could really impede your progress. So what are we going to cover today? What is the objective? You know, so we'll talk about a little bit about what is this governance which I kind of kind of alluded to as, you know how do you come up with a way for all of us to be good citizen of this new distributed world that we're going to live in. So we'll talk about that how to establish how to think a little bit about that kind of governance framework and then we'll look at, you know simple but kind of descriptive examples of just kind of open policy agent what it does and kind of the legal language and also we'll address a little bit about gatekeeper which is something that has been in the making for some time now but it's kind of, you know becoming more productionized than before. So and we'll even look at some examples of policies and Kubernetes that you can utilize. Now, instead of making this just kind of a very abstract actually I'll go through a true story that happened to us at Magalix and kind of how we went through and use that as a motivation to kind of really walk you through kind of the workflow of the process and how we ended up doing what we do and hopefully you learn from that as well. So one day, one of our SREs had this message in Slack there was this workload running in our dev environment and he didn't know what it was. He was doing some changes and he needed to change something and he needed to change the configuration of this thing and he wasn't sure who owns it because he needed to talk to them as this is, you know is my change again affects you, you know and I'm not, I don't know who this thing is and it took a bit for people to reply. Now, could this have been a malicious thing? You don't want to wait too long to know what's going on.  You know, who should we contact? Maybe it is something in production or something of a critical.  So that delay, you know causes problems and you really want to get an answer maybe to something like this very quickly. And we thought about it a little bit eventually we knew who it was and it was somebody prototyping something as part of some new feature we were working on. So we figured that out. But then, you know, what do you want to always learn as part of any operational excellence, you know framework you have, you want to be able to prevent issues you reflect on those sort of problems. It's just a simple one but you can think of this as a security violation somebody found somewhere and I need somebody to fix that very quickly. You know, should we then block all developers and force something like everybody should put an owner name maybe for every application that is deployed on our Kubernetes infrastructure. You know, how are we going to do this? Are we going to force a PR and like have the SRE manually review all of this? You know, this has got to be a better way. And usually the better way has to do something with automating the process. So let's call this the owner label problem. Like we thought about it like, what if we have a best practice or a standard in our organization that says, you know every workload you deploy must have a label that has the owner name or that could be an email of the team that owns that service. So at least we know who to contact or who to talk to. You know, you can think of, you can extend this. You know, maybe you should have also a link to the GitHub where the service code at. Maybe you can have a link to a wiki page in your internal organization, subscribe, maybe something a bit more about this. So if somebody want to learn about it. So whatever it may be, and you can extend this just simple idea of owner label. But at the end of the day, how do you enforce this? How do you make sure everybody's playing a good citizen without blocking the productivity of, you know the engineers want to move fast and move quick but still you want to provide this level of kind of checks and balances to ensure a common standards across the organization to make really your life easy down the line in production. So this is really kind of a governance problem in a way. So when we think about it here is, you know it's the idea of policy as code. Now we can write a document. We can have this in the onboarding. We can have emails going out about this new policy but that's probably not a good scalable sustainable solution. You really want to think of just like infrastructure as code, you want to have policy as code. So what is governance at least in the way we define it here is it's the ability of the operation team to verify and enforce certain policies and standards across the entire organization or maybe a subset or a certain cluster or a set of clusters or a set of workloads that meet certain criteria whatever it may be your ability to enforce certain things and be able to do that automatically in a productive manner is what I'm referring to as the governance framework. So there's kind of three things usually when you want to think about establishing your policies. First, what is your target? So in this example, we have in this particular case that the target here would be really workloads, any workloads. So not ingress, not services, not volume just kind of any object that is a workload. So controllers, you can make it more specific. Maybe you care about stateful sense more than certain other things or maybe certain objects and Kubernetes that have a certain annotation, whatever it may be you need to kind of for any given policy you want to kind of define what is the target and then you want to define the actual policy. The policy is a set of rules you want to enforce in our particular example is anything that is missing an owner label is a violation of our policy. So I want to check, I want to ensure that every workload controller being deployed to my Kubernetes cluster has a owner label. The last thing you want to think about also is the trigger. How do you do this check? Is it something you do once a day, once a week kind of on a schedule type? Is it something you do at deployment time? So in your Kubernetes like something like admission control for those of you familiar with admission control. So you do it at deployment times you prevent the deployment any deployment of a workload that violates this policy is that when you want to enforce the policy or could you do it even earlier? Like move into the left thinking about your build time your CI phase or even your commit your get commit phase and can you enforce it at that level? The more you move to the left the better because your developers your engineers will get kind of the feedback early and hopefully solve the problem before it's, they're in the middle of a deployment and now things are failing and now they have to go modify some code or update some files and go through the CI CD again. So those are the other three elements you want to think about when you're thinking about a policy. For our case, I want to enforce this on all workloads in our Kubernetes cluster for our dev and product environments or two clusters. And the policy is I want to make sure every workload the spec has a label called owner with some text inside. And the trigger we were fine every 24 hours just once a day I just want to get a report about it and then I'll go harass the people and then maybe later we can make it a little bit more dynamic and maybe enforce it at build time which would be awesome. Fail the build if something violates this policy. All right, so let's talk about open policy agents which is the first thing that should come to mind probably when you're thinking about enforcing policies in an automated manner. So the open policy agent is an interesting so it's part of the CNCF, it's a CNCF project and I think it's they just file for graduation. So it's a great tool to allow you to, you know your organization to divine these custom policies and be able to run them. Now that the OPA itself is just kind of like a policy execution engine. And it uses a language called Rego which is kind of the policy language. So we'll talk about it in a bit. But truly what it comes down to is let's talk about kind of the case when you're deploying things so you make a request to your Kubernetes you're trying to deploy something, update something what let's say an admission control case that change will go and your OPA could intercept that and then respond evaluate basically the policies associated with this change and then enforce say, you know, deny or accept or deny this change. So this is roughly kind of the paradigm. The idea is I have a policy and I have an object that is being changed and what the OPA is kind of will take these two and tell you yes or no, deny or allow. Is this change, does this object check the boxes, check the list for this policy with these policies, maybe have a list of them? Does it violate any of those? If not, then you're good to go. If yes, it violates them, that's a deny. So that's really just kind of a high level the essence of it. But how you deploy it and the architecture of it is something we're gonna talk about in a bit because there's different ways and that kind of depends on the triggers, right?  So that by itself doesn't decide really the other things like the triggers or the policy. The policy you're gonna define with ego, the ego language, that where to enforce it is another thing we're gonna talk about and then the triggers is when to enforce it is the other one. So let's start with just the policy itself. Now, how do you describe the policy using kind of the OPA? Well, there is the ego language, which is a declarative language and I just made a very simple here policy. Again, I just checked that the label exists. It doesn't even check what the value is. You can make it more complex. There is a lot, I won't be going into details of how ego works or some of this. There's a lot of resources online to help with that but I just wanna get you the flavor and just put it all into kind of one end to end framework so you can see how it all works. In each piece of this, you probably could dig more. So this is the simplest one. Now ego is just basically statements like assertion statements that you evaluate to true or false. Even an assignment statement is evaluated but it gets evaluated to true by default but everything else is almost like a conditional statement of a sort and then it gets evaluated to true or false. Everything is true, that's a pass. Something is false anywhere. Well, that's false. The whole evaluation of the policy becomes false. So if you look at it here, there is input.metadata.label.  So the OPA kind of handles that and kind of assign the object that comes to this input variable. And then this is the spec of a workload of a controller object in Kubernetes. So dot metadata, dot labels. And then you can access, you look at basically for an owner label inside of that YAML JSON structure. And if it's not there, you know, that will evaluate basically to denial. Let's actually show you, if you haven't, there's something called the ego playground and it's available there. I think the URL is regoplayground.org. I should have put the link here. It's easy to find out, but let's have, here's the policy to your left as you see. And then to the right, I just put an example. Again, this is not the full spec of an actual object, but this is just kind of a part of it. And you can see, so this one doesn't have an owner label, has another label. So deny equal true because this statement ended up evaluating to true. So when you click evaluate, you'll see kind of the output here deny is true. This is another example where there is an owner and now there's nothing coming back. So what you see here is just kind of how would the OPA would evaluate these policies? And honestly, it's up to you the deny versus the allow how you wanna structure because you will interpret the result. And we'll maybe talk about the deployment. So you kind of maybe see how would you deploy this? So how would you interpret this result? So there's three ways really to deploy kind of OPA. One is just use the go as a library really, just use the go module and write your code. Just like this example here. So there is a rego.new and you create a query and you can parse and evaluate assuming that ego is in a file called example.trigo. But you also could deploy it. OPA comes as a container and you can deploy it as a container inside of a pod. And then your application can call that pod, maybe pass it the object it comes and then use something with it. And OPA also could be you could configure it. So it receives just as a full pod by itself, like in the third example here and it has its own URL and you can create a service for it and you can just kind of OPA as a service, you maintain that. Now what we decided actually in our case is we just went with the go library because we wanted really full control of creating our own service and getting the object and interpreting that result, the deny. We wanted also a standard within our organization of how we want to write those policies like those assertion rules. The word deny is just a rule and you can create whatever you want. And it doesn't have to be like we said, it really works in any object. If it's adjacent, it could work on it. It doesn't have to be a Kubernetes object like in our particular example. So it definitely extend beyond the one example I'm showing you here, but there's a lot of options. But the ones we went with at least for our case because we wanted to really have a kind of a standard within our organization to do this and also help even our customers with this. So we went with basically writing our own service and using the OPA go library to do this. But again, at the end of the day, you just need the policy, you need the object and then the OPA is just a policy execution engine which is no magic to it. How you write the policies, how you want to interpret the result, that's basically up to you. So this is something that is not part of OPA itself. It's just the policy execution engine. So how do you manage your policy? So these are some of the things that are missing with OPA, the vanilla OPA. Where do you put those policies? How do you manage them? When do you run it? Somebody has to call this. Even if you're on OPA as a service, you don't write your own. Somebody has to call that service to do this. And that gets us maybe into some other ways to trigger this, which is really Gatekeeper. Gatekeeper is kind of an extendable parameterized policy library that I think now is v3 and it's still in beta, but there's a lot of contribution from a lot of people and I think it's picking up some good steam here. And it tried to address some of those shortcoming or basically adds to the OPA so you get kind of the full framework of a policy as code. Cause like you saw there, you can write a policy but where do you manage it? Where do you keep it? The other thing is when do you trigger it? Who's gonna do the triggering now? Like in our case, we ended up writing a service and we decided to do some events and when something changes, I will call the service to kind of validate, verify the policies at work. Is that change that somebody made violates any of our policies or not? So Gatekeeper really tried to address this. And let's see if the next slide is the example, but let's just kind of the architecture of it. So it uses OPA, you think of it as a layer on top of OPA. It will run like an agent in your cluster and it will register itself as an admission controller. All right, and what about the policies? Well, the policies will be custom resource definitions. So they have the resource definitions, not just the rego now. So you create a policy, which is gonna be its own YAML kind of object that is a CRD.  So that's how would you manage those policies? And then Gatekeeper will register itself, a webhook with the admission control. So any changes that happens in your cluster, you will get kind of an admission request or an admission review request. So this is the triggering mechanism like we were thinking about the governance framework. And then we'll talk about actually how you do the targeting with this, but basically the Gatekeeper will get any change from the API server and the webhook and it will run the policies, the relevant policies, and then we'll give you an admission review, deny allow. So what does these CRDs look like? Well, there's, you basically create a template, which is this kind of example on the left, what's called constrained templates. And all you see there is really the rego. So the missing labels or what have you. And also you can define variables and that's a very powerful aspect because again, that's what makes this a template is you write it once and then you can verify because sometimes, I know I call it owner label, maybe you wanna call it something else or maybe I have different, in the databases, I wanna call it something or my services for application or workloads that my team owns, I wanna call it owner, but maybe for third party workloads like databases we just get from third party, we wanna use a different label. But it's really the same policy here, the same template and you can have what is called constraint which we have an example here on the right where you kind of plug the variables you want, the exact value for the variables and also this is where you get to define your targeting. So if you look to the constraint here on the right, the parameters here label is owner and you can see I'm for this particular case, I have it target kind deployment. So deployment objects is where I gonna run this policy. So you see here now the gatekeeper is providing the targeting with what you see here and also a decent way to manage those policies we talked about. So it's not just creating the policies using the rego but also a way to manage those policies. And because it's registered itself, you deploy it as an agent and it registers to itself, now the triggering will be anytime something changes you will get a trigger. And gatekeeper will do the magic for you, we'll find the objects, whatever the object that's being changed we'll find the policies that matches that object and then we'll run them and then we'll kind of give you the result, the deny allow kind of admission response. So you can try like this example here where we have something that was missing the owner label and you see just when you're trying to do a bad deployment you get this error from server denied by must have owner and tell you they're creating this admission whip hook and the gatekeeper and then so on. I don't know the time. So I just talked about one example like this owner label but obviously there's other ways, other things you can do you can create policies for, you can check for readiness probes and liveness probes, that's just a good practice. Every workload you have, every container you have should really have those liveness probes and readiness probes, the services. You can obviously also enforce certain just basic security like hygiene, good hygiene like allow privilege escalation must run as non-root. You can maybe enforce certain things like workloads all your services should have at least two replicas just for full tolerance of one fails or one node has an issue, you have another replica. Affinity, pod anti-affinity is another thing making sure your pods, your multiple replica of your workload is not deployed to the same node. So that node goes down now your whole service goes down even though you have two replicas but they both ended up in the same node, that's not good. Role binding, container images, check in, you know, making sure you're only getting your images from trusted repositories and so on and so forth. These are just kind of very simple basic examples but I'm sure you can come up with more you can come up with your own as well that kind of makes sense also for your own organization. This is a bit continuing with kind of the gatekeeper because once you do the run it actually the status part of, you know when you do describe for your Kubernetes constraint object you will actually see the status. So to kind of give you an audit and a status especially when you create that constraint for the first time, it will kind of do a quick run for you and show you all the things that were denied or would have been denied. Basically objects you have that already are in violation of your policy. So to kind of really wrap it up before I even talk about kind of, you know let's just finish with this. So there's also metrics, you know, gatekeeper really I think take kind of the OPA and kind of the concepts of the legal like these basic building plots and try to put it together and provide this tool really that will help you then implement this kind of governance framework that we were talking about because the idea is you need a way to target the changes of the objects you wanna enforce your policies against and then you wanna define your policies which means also you need a way to manage them and put them somewhere and then you need the way to trigger those policies to trigger those checks. Now gatekeeper does it just kind of with admission control if you need to do other things. Again, OPA is a open source but also they have a nice go library so you can just use the code and write your own service which is kind of the case we've done. We also look it into gatekeeper it's still in kind of the development phase really that is I know some people may go around adopting but I think the constraint and the template the constraint template and the constraint framework that's created definitely move us forward in that path. So now you can see kind of the end to end story here you have a policy you discover some issue in your environment where we needed everybody to play as be a good citizen of this distributed environment in this case just having an owner label or whatever other operational excellence best practices you might have, you take that you codify it with code. So using Grigo to make your policy as code as opposed to just kind of like an email that somebody would send or something you put in some like onboarding handbook that probably nobody will follow. So now you are enforcing your policy as code which really give you this kind of automated ability to ensure kind of good hygiene, good operational excellence good stability of your production environment without putting a lot of strain on the developers and slowing them down and impacting kind of the agility of the whole process. And thus you as both dev ops kind of the devs and the ops really focusing more on the interesting problems of the business and really focusing on the value for the things you wanna do for your business as opposed to kind of fighting and enforcing and who missed that and the dev, the ops team zone just like code reviews or PRs or trying to really chase developers to make sure everybody's following the policy. So with that really I concluded kind of the webinar and I'm happy to answer questions it would just kind of brief high level overview kind of looking at three ago OPA and gatekeeper and how can you kind of put them together with kind of a specific kind of problem just to motivate the discussion. So with that, thank you and I return in the back, Gary. Thank you Ahmed for a wonderful presentation. We have plenty of time for questions. So if you have anything you would like to ask please feel free to drop your questions into the Q and A box and we will get to as many as we can. Do we have anyone at all? Many people probably haven't slept since last night. Everything I guess was clear this month. Okay, so we have a question here. Could you give a bit more background information on gatekeeper? Yeah, so gatekeeper. So it is also a CNCF project. It is really built on top of the, it uses OPA. So it is open source and you can definitely find it and they have nice tutorials. But what it is at the end of the day is an agent you deploy and it registered itself as an admission control. So what admission control is, if you're not familiar with that, it just the API server allow it to register a controller with it and register a webhook. And when something changes it will send you kind of a request to the webhook when you respond to it. So it is called an admission request and it passes you the object that is being changed. Kind of the old and the object and you get to decide if you allow this change to happen or not. So how would you allow it? Or how do you evaluate it? Well, you have the object coming in. Now you have maybe policies defined. They are being defined kind of like those CRDs like the constraint template and the template like the example we show. So the gatekeeper will automatically be reading those and watching them and evaluate those policies against the change. And the admission response is either basically allow or deny true or false in a way. And if it is false, it will kind of fail the deployment. So whoever is running the kubectl command or what have you will see a failure like the example here we showed. You will get kind of an error like this. So if I'm doing this deployment and basically I have this YAML spec and it doesn't have the label. When you run the kubectl command, the API server obviously everything go to the API server. Notice that somebody is trying to change some object and it knows that there is this gatekeeper who's registered as an admission controller. So it will send it a request, that admission request basically ask it, should I allow this change to go through or not? And then because I have a constraint, constraint template that I constraint that has this owner label policy, the gatekeeper will evaluate using OPA will evaluate that policy against this change and denied it because it doesn't have the owner label. So kind of high level, that's really kind of what's happening right now. That's what gatekeeper is. It is currently v3, if I remember beta, I mean, there's a lot of people contributing to that project. So we'll kind of see how it matures and when it becomes kind of production ready. I know some people probably already have it in production but there is some really good active development on it. So I encourage people to try it out. Sorry about that, I was on mute. Do we have any other questions at all? So if no one has any other questions, I think we will wrap it up. I wanna thank Ahmed again for a wonderful presentation and for taking time out of his day to join us today. And I wanna thank everyone for joining us today as well. As I said before, today's recording and slides will be on the CNCF webinar page at CNCF.io slash webinars. That just about does it for everything else. Thank you all again for attending. Everyone take care, stay safe and we will see you at the next webinar. Take care everyone, thank you.


Well, as of this afternoon, the 2017 Beltrami County Fair officially opened its gates for the season. This year is billed as the biggest and best Beltrami County Fair ever. The Midway kicked off the fair with arm bands for $25 and food vendors started cooking up all the fair favorites. Judging for all the exhibits began today and will continue throughout the fair, improvements have been made throughout the fair grounds so that everyone is able to check out all the fair has to offer. It's one of the busiest fairs we're looking forward to having. Our commercial space sold out especially early this year. All of our camping spots have been sold out. We've never sold out on camping spots before. Our carnival seems to be bigger than it's usually been. And so I've been to six fairs this year prior. They're all very busy this year. They're all building up, they're growing up new buildings and they're really upgrading a lot. Their interest in fairs is just increasing. While out at the fair, be sure to check out our Lakeland Public Television booth. We have some free treats and giveaways for the kids. It's also a great opportunity to enter our crews with the news giveaway. I'll be up there Saturday afternoon from noon until four. The Lakeland Public Television booth is located in the commercial exhibits building. If you've enjoyed this segment of Lakeland News, please consider making a tax deductible contribution to Lakeland Public Television.


Okay, move your faces. Okay. Here and here. You can start the recording whenever you want. Go ahead. Hello everyone. Welcome to the Jenkins Weekly Infrastructure Meeting. We are the 8th February 2022. So today, as attendees, we have myself, Hervé, Marc, Tim and Stefan. Okay. Let's get started with the announcement. So last week, we didn't have a weekly meeting, but last week, the weekly 2003-03 was released successfully, as far as I remember. At least it's running in production for us on Infrascii. So thanks for the release. This week, there isn't any weekly expected, at least not today and not tomorrow, as far as I understand, because there will be a security release on weekly and LTS. I don't know if we should trigger a weekly Tuesday or Friday. I don't know, Tim, what do you think about that? I know. So there will be a weekly tomorrow. It will be 2.334, but it will be a security release. So it's 2.333 plus just the security upgrades or security changes. So no normal weekly expected. I don't need a security one this week. Yeah, no form modernize. No form modernize. Okay. Yeah, form modernization first arrives in next Tuesday's release, right? Okay. A huge PR that was made from Monday. And Tim, any opinion, just for my while we're here, any opinion on should we include system D, transition in that, or should we wait a week after that? I think we're probably okay to include system D in the form modernization one. Okay, great. Thanks. Yeah, I marked it as on hold as I didn't want Bezel to merge it to master. Merging it before the security release would be disastrous, right? So that's one. Yeah, good. Okay. About the next week related also, there will be a set of security release for plugins that should happen 15 of February. So I took the responsibility to say we will delay the weekly to the 16th of February to Wednesday. There shouldn't be any problem releasing weekly on plugins the same day. However, given that it's sensitive because security, because we messed up the automatic weekly last time, I would prefer to have one thing per day just to be sure it's clean to avoid any rush, because if security is late, or if we have mirrors that happen at the mirror synchronization at the same time, there might be what be of yours, resulting on security plugin release before the announcement. So there should be no problem, but better safe than sorry. So the weekly with system D and form modernization should happen next for the Wednesday, 16 of February. Tim, that okay for you since as release officer, that's probably your domain to make the ultimate decision. It's just weekly, isn't it? Yes, right. No LTS is this week, so nothing but LTS. Don't care about weekly really, just whenever it comes. Great. Okay. Just to say it's system D, not system V. System V is the old unit, or it's referred to something. Transitions. Is that not right? It's maybe okay to include system V in it to system D. Ah, okay, sorry. I misunderstood. Sorry, I was skipping my bad. Just poor phrasing from the person doing the typing. I fixed the phrasing. Hopefully, it's more readable now. I thought that's why you were writing initially, but then I saw the end of it. Yeah, it's the non-English native era. That's misunderstood. Also just poor phrasing. So there's nothing to do other than bad phrasing on my part. No problem. So these are the three main announcements. Do you have other announcements to make? Okay. So let's proceed on the tasks and notes. First, top priority about the two security releases. So this week, tomorrow, we will have cores only. Remember, don't merge anything this afternoon or tomorrow until the security release is done. At least nothing that could have any impact on release.ci or ci Jenkins.io, please. This morning, we took the opportunity to update release ci infra ci ci and trusted kernel plugins, everything that was available this morning. So tomorrow, the security team will have a clean state to work with that would avoid them having a zillion of plugins to update and they have to manually select. Most of the time, they don't need to, but yeah, at least we have a clean state for them. Once again, the weekly automatic release triggered automatically. This time, it's no one's fault. Compared to last time, it's because we had a short notice from the security team that they were willing to have a core release tomorrow and they were going to stage it Monday. At least I received the notification really, really early, really late. So yeah, but just to be sure, thanks for the review. We disabled the management of release ci on Kubernetes. So the proposal is that we don't put it back on automatically managed until the security release is finished and we will have to do the same thing next week. Just to be sure there isn't any unplanned reload of the configuration on Kubernetes that will enable again the job. So is there a way we could change the Kubernetes definition such that the job starts is defined disabled during these periods or that's not not really workable? There is a contribution to be made on job DSL so that the attribute disabled is exposed on the job DSL configuration and used. Okay, so right now you can't create with job DSL a disabled job. Yeah, exactly. Part of the code is already there, but it has to be exposed as an attribute from the configuration. Yeah, we get to do that. But in the day, we don't really want any change to release.ci. We don't want anyone updating plugins or the image or anything while we're anything that could take it down while we're about to do it. However, I also had a smart idea that could be complementary to that, which would be changing the pipeline for the releases to check for a log file or not. So that would mean if anyone from infratim or security team commits the log file on the repository, when the automatic weekly trigger starts, if it sees the log file, it will stop immediately with a message saying, I got a log file, I should I'm not expected to trigger weekly. That's a simple one that will be manipulated as code and we won't have to care about any Kubernetes reload or job DSL. So say that again, which log file? That sounds interesting. Okay, sorry. Perfect. No, so it was it was a file that explicitly says this system is locked. And because this system is locked, do not allow the job I see. Thank you. Okay, which will be full as code only restricted to the release pipeline. So releases.ci.genkins.io could still continue working and being updated without any risk. Right. So that should be a complementary process, but we still need job DSL contribution because disabling jobs is something we need on numerous places. Okay. So that's why I wanted to mention that part. There is an issue. I've put all the links. And Danielle yesterday did the part of the staged the release part, not the packaging part, only the reason generating the war and signing it for the weekly, it worked perfectly.  So that was the same code. So that's fine. About LTS, still the same issue, which time I make a change to the master branch. I forgot about, I don't know which branch I have. I don't understand that it's really painful for me. So we ended up on the same issue as we had two weeks ago. The jar verified missing from the bin because using the doc image older than the one weekly. Just to be sure, Danielle did the manual, the verification manually, like we did two weeks ago, I just ready to continue the packaging. But just in case, thanks team for reviewing that, by the way, I've merged these changes from the main branch to both stable 2.319. So the LTS branch and the security new branch. Because as I understand, there is one branch for each patch of each LTS on security releases. Most of the time, they build it from the stable branch. That's too much for my brain. So I've, I did that these two places just to be safe. If I forgot any target branch, don't hesitate to open the pull request and mention. I suspect we already have a target branch for stable dash 2.332.1. So maybe worth checking to see if. Is it for the patch on the weekly? No, that's for next LTS for the March LTS. Okay, so that means the error also happened on that one for sure. And I assume that Daniel, oh, it was using an older version of the pipeline set that is still working by any luck. It has that that release hasn't been done yet. So we wouldn't we wouldn't have detected the failure. It just I suspect there's a stable dash 2.332 branch out there. Okay. Who is going to do that part that staging release? Is it us or security team? And sorry for clarity, there won't be a security release of 2.332.1 because that's next LTS. That's the March LTS. Okay, okay. This is just me for warning that I think the branch already exists and needs the cherry pick. No, nothing more than needing the cherry pick. Okay. We have to take care of that. Can I defer that to someone else than me just to be sure that that's knowledge is shared? Sure. So I'm happy to take it or I could I could pair with Stefan and he and I could do it together. It might be a good experience for the two of us to pair. With pleasure. Okay. So how about if Stefan and I take it? Okay. So let me write this down. Yep. Cool. There are stills. So yeah, that one was important to have in mind for tomorrow's release. For tomorrow, I don't have any other elements. Do you have any other points for tomorrow's core releases? Okay. Just a word for next week that will happen before our team meeting. That's why I wanted to mention it. Trusted will be needed at all costs because it's only plugins. Release.ci However, let's keep it safe and CI Jenkins.io, of course, because they will need to update the plugins if we have some. I haven't seen the list of plugins impacted by the CVS stuff. So let's assume CI Jenkins.io will be, especially since Vadek asked for the status Jenkins.io. For both releases, I need someone. I can take it. Someone can be me, but someone need to update status Jenkins.io to say that CI Jenkins.io might have some restart and unavailability tomorrow and next week, the 15th. Is there someone volunteering to updating status Jenkins.io? For the whole day? No, no, no, just for it's usually it's a our history says it's about a it can be anywhere from a 15 minute to a 45 minute downtime. And Stefan, if since Stefan, I are going to be cherry picking on a branch, why don't we also take that one? Because that way he and I can do a how do you do a submission to status? Looks good. Thanks, folks. So let's so that will be for tomorrow's release and next week. So you can do this and pull a request. Good. Yes. Okay. Is there anything else about the security releases this week or next week that you have in mind? No, okay. Thanks, folks. Next priority topic, Digital Ocean, work in progress by RV. Irvade seems that locally you successfully created the resources. You were able to run the terraform process and fix it on numerous levels. It seems that the read me that I originally created for AWS terraform management needs some a bit of revamp or update at least and the file structure. We had good exchanges about how to build a stronger shot pipeline library for terraform management and testing and stuff based on the issue that they had locally due to, let's say it's the youth of that terraform stuff. I understand, Irvade, can you confirm that you are the CI steps, meaning making the CI jobs working? It's created, but you have to make it work so it managed Digital Ocean cluster. For now, my big issue is the credential for the backend config file, which are turned correctly interpreted. It puts the environment variable name inside the backend config file instead of the content referenced by the environment name. So I have to manually upload a new version of them each time the team or mark does it ring the bell when specifically saying specifying a credential of type file from jikask? That should be a base 64 from environment variable. And what happens is that the environment variable is put as content of the credential, so it's correctly encrypted and decrypted when pipeline use and load the credential. But we have to set the multi-line content by ourself inside Jenkins through the UI because it doesn't work from jikask. It should work in jikask, but it's easier if you just avoid this and just use string variables. I'm not sure exactly what's in this file, but normally if you just do something like an AWS or Azure, you just do tf-r-m-i-d equals and then have the key as an environment variable. So like with the Azure Terra from provider, you can provide files or you can just provide environment variables and it's not simple. Not for the backend, you must use a file for the backend sports because most of the key are or you have to define all the you could define the keys inside the backend.tf file, but the Terraform in it needs a file in order to have a lock. Yes, it's to have the lock on the backend. Is this for the like the S3 bucket or something else? Because looking at the digitalization talk, so it's very sturdy. And I'm surprised it's the same mechanism as AWS as far as I can see. You can supply it on the command line. Yeah, I can. I think I can. But I was thinking I can inject the variable via a little bit of script in the beginning of the pipeline if needed. It also says that you can do environment variables too. You don't have to use a file. We still have from 1.1 maybe, but not with 1.0. We haven't been ready to latest minor. I don't think the stuff doesn't tend to change very often. I mean we also even if you can't use environment variables at the very minimum, you can supply it on the command line. You can go Terraform in it's dash backend dash config and then you can supply a set of those. Also another clue is that since the content of the backend config for the actual Terraform and the digitalization are almost the same, there is only one change, one key. The idea would have been to reuse the existing credential and then only define the key on the backend.tf of digitalization because it's not a sensitive data. Only the key is just a namespace for the Terraform state inside the bucket. That was the thing I had in mind. However, we still have an issue on our setup and I don't know if it's gcask, if it's sops, if it's the way we inject the contents, but we have exactly the same sops data for both credentials and here you see the configuration inside the cask configuration passed to the elm values and for both we use the base 64. In the zoom set, I have put the error log I've got in the Terraform pipeline. Where we can see the contents and the file contains only the variable instead of its contents. So that one line 154 works as expected, that environment variable has a multiline string content coming from sops decrypt it and it's correctly base 64 hashed and it's set as a credential that works as expected. But the digital option one doesn't work. Did you say it's just not been substituted as if it's just got the variable there? Yeah, it's got the content of backend config in the pipeline. It contains the variable name, literally what I have on the line of my screen share. Okay, I mean it doesn't sound, it's probably not jkask related. I'm pretty sure. And I've checked the variable name, in case it does a typo, but it doesn't seem to. I think jkask will substitute it, it's been in the string if I can't find it. So it's probably, I'm not, I think it does that. So it's probably... My understanding is that that variable is not defined. Yes, so it's literally jkask just take it because it has different three ways of using the base 64 method inside the environment. It can use read the content of a file and base 64 it or a string or it will have a reference variable. So there is something weird. Sorry, we sidetracked, just in case it was a well-known behavior, I don't know. So we have something weird and we need to change or to fix that because it's slowing down our way. I feel like it'll be a type or somewhere. Yeah, I'm too bad at that. For sure, but yeah, we can find it. But yeah, outside that credential issue, almost there. So good job, RV. Next topic, unless you have a question or something else. So update CLI, so great work, Stefan, on adding more update CLI tracking on more dependencies. We have HashiCorp tools, which has been a bit too recently and the most interesting part is Kubernetes management is now tracking the Amei for the EC2 agents that we recently added on release.ci and infra.ci. That should allow us to build using Docker engine on virtual machines and that should allow us to use the same labels for both CI Jenkins IO and our Kubernetes Jenkins Controller. So in the future, we should be able to shrink the pipelines to use only Jenkins file and migrate trust it to release. That will open this road. So thanks a lot for that work, Stefan. You're welcome. Private AKS cluster. I haven't done anything. I was in day off, but that's my top priority for this week. Focusing on the Groovy pipeline library for terraform management and then re-initialize the Azure repository like the digital ocean that is working on. So we can create a private cluster and move at least in FraCI and release CI with behind the VPN. We had a word error on service mirror Jenkins IO. The error was said a bad certificate except that we never used mirrors with HTTPS. That was the whole reason for migrating from mirrors Jenkins IO in virtual machine in Amazon to get Jenkins IO with mirror bits inside the Azure to have TLS because the tooling used for that legacy mirror Jenkins IO web service wasn't supporting HTTPS. So we don't know how the probe, the Datadog probe could have worked before that day, but it was eroding. We removed the probe, but that starts the question of decommissioning the service mirror Jenkins IO. There is an issue to be created. Thanks, Tim. That's a good point that you're on the line. That's worth communication, at least a blog post on the Twitter before really starting to decommission. And as I've been underlined also, we might want to search for the references of mirrors that Jenkins IO inside our own code base in Jenkins Infra, at least test RNS and pipeline libraries and change them to get Jenkins IO. So at least they will use the correct service. Not sure. There are several references to mirror the Jenkins IO on Jenkins IO organization too. I don't know if we should put a CNAME alias to keep mirror the Jenkins IO or to change this URL there too. That's a discussion that would have need to happen inside the issue for that decommissioning, but yes, you're correct. Should we change on client side, server side, but another word on rating the Jenkins IO. We have Jeremy Playout from Cloudbees that started to work on to take again the work that gave in on maybe others started one year ago, at least the goal is to migrate the service rating Jenkins IO from the tiny virtual machine on Amazon where it's running today into Kubernetes. So it took the work that gave in pointed at us to have an Elm chart that he was able to run locally. There is a pull request open that we have to review. The next step will be to prepare the installation of that Elm chart on our pod public cluster. Attention point for the infra people. Rating need a database. It's a PostgreSQL database. I don't remember where it hosted. I assume it's on AWS as for today. Yes. Need to be checked. Okay. Yes. So we might want to create a new PostgreSQL database inside Azure. At least manually and I will import it on Terraform later when it will work, but we need a PostgreSQL and there will be a migration phase. We'll have to dump the SQL from AWS and insert it inside the new database. The reason is because accessing a web service on Azure, accessing a PostgreSQL database on AWS, not only it will be slow with latency, but also we'll have to tweak finally the IP allow to connect to the database. Given the tiny amount of data inside the database, better to migrate it directly all on Azure and remove all these resources on AWS. What do you think? So thanks Jeremy. Jeremy is out, so we will continue. It's not a priority topic, but on background, that's interesting. An external contributor doing that for us. That's really nice. It's four, so I propose unless you want to bring another topic that we delay the other topics to next week. Is there any prior topic that we forgot about or do you want to speak about right now today? Okay. So thanks a lot everyone for your work. I'm going to update and prepare the note for next week. Move everything we didn't treat it today and publish everything on this course once the recording will be available. Thank you. Thank you everyone. Have a good day. Take care.


I'm going to show you how to set up multiple email signatures so that you can send an email to students, to parents, to colleagues, with a different signature at the bottom. So sometimes you want different information to show for parents than maybe you do for students. So it's really simple to do. All you need to do is go to Gmail and then go up to your gear in the top right, up to your settings gear, click on See All Settings. On the screen when you scroll most of the way down, about three quarters of the way down, you will see Signature. Now you can see I have three of them set up. My basic signature just has my name and a phone number, my job description. My full signature is what I use normally with everybody that I work with in my district and then an out of district one. Now the difference would be like in my district I allow people to book time with me using a Calendly link. Out of district takes that Calendly link away and just adds my portfolio information. That way people outside of my district aren't booking me. So depending on my audience depends on which signature I choose to use. So it's really easy to do. All you have to do is click this Create New button. It'll ask you to name it. I'll just call this my demo signature. Click the Create button. And then on this right hand side over here is when you will now create it. So maybe I can just say Jeremy Badner. And you could put any of your branding here or whatever you want. You can insert pictures and images. And I also have a document on my blog on how to make creative signatures with images. So if you need that, that's available as well. So once you've got your new signature created and you can see it's called demo signature here, you can also set which is your default. So for all my new emails, I use my full signature and which ones I wanted to reply with. I could change it to any one of those that I want to use. So maybe I wanted my demo signature to be the one that I want to use all the time. But I'm going to keep that as full signature. So when you're done, you've got your signatures listed here. You scroll to the bottom, hit Save Changes. And now when I go to compose and I create a brand new email, whatever my default signature is is there. But that doesn't mean I have to stick with it.  And when I click that, it says which signature do you want to use. And here's that demo one that I just created. Maybe I want to use my no signature to take everything away. Or maybe my basic one was just my name. So what it makes it easy to do is include information you need to the right audience. So you can choose whichever signature you want. And then when I send, it will go with that signature. It's as easy as that.


I am part of the 2019-2021 batch of MSc students who started school here back in October. And it is now January, so it's been about three months and I do feel pretty settled in here now. So I thought it would be a good idea to take you guys along with me so you can see what it is like in a typical day for an IHG Dolph student. So once I get done with my breakfast here, I'll be taking you guys along with me to school and we'll see what I get up to while I'm there. It's just like a five or seven minute walk from IHG. So every morning I'm able to just walk to school, thankfully. I do have a bike. There's lots of bikes and scooters around, but I don't need to use it every morning to get to school. Inside of the school, they serve breakfast and lunch here. And here, they've got free coffee. The Environmental Science Program and my specialization is in technology. And so first period today, I've got wastewater treatment and recovery. We're off to period two, which is microbiome. We're about to have our lunch. Yum. Well, I brought my lunch today, so I'm just going to heat it up in the microwave but there is also the option to buy a lunch at the restaurant. It's half the day off now, which gives me time to probably hit the gym before it gets too busy and then I will be going to the supermarket to buy some groceries. And I am really in luck today because as you can see, it's sunny and not raining. It's gone and changed to go to the gym, but before I hit there, I thought I would give you a little room tour since the lighting is good right now. So here we go. So this is what the biggest part of my room looks like. I've got my bed right there, my closet provided with this cabinet here as well. Over here, I've got a little dining table area and I'm lucky because my room looks into the courtyard back here, which is beautiful, especially when it's sunny out like it is today.  So I've got a laptop here, which was rented to me by IHE. All students get that laptop and then my own personal laptop as well. Moving into my kitchen. I live in a self-contained unit, so I do have my own kitchen. It comes with a single induction cooktop, a little fridge and a couple cabinets. Pretty simple. Let's see what I've got in my fridge. It's looking not too bad right now. Not a ton of stuff in there, but I also have a freezer space available in the common kitchen area. And over here, I've got my shoe rack with four shoes over there. And this is my bathroom. Pretty small, but it works. Just toilet, shower and then from here. I should also add that every room does come with basic furniture such as a bed, a closet, cabinets, a desk and a dining table. So you wouldn't have to go and get any of those things yourself. Alright, now that we're done with the room tour, let's go to the gym. It's 4.30 now and as you can see the sun has started to set, which is pretty normal here, especially in the winter time. And I'm going to get home, get cleaned up and then make my way to the grocery store. So I'll see you then.  Alright everyone, so that's a wrap for today's video. Thank you so much for watching. I hope that it gave you a little bit of insight into what a typical day is like for an IHG Dolft student. If you liked it, please give the video a big thumbs up. If you've got any questions or comments, leave them in the comments box down below. And share this video with your friends or anyone you might know who is looking into studying abroad for their graduate studies. I made this video in collaboration with IHG Dolft, so don't forget to subscribe to their channel over here. And if you're interested in seeing more videos on what it's like to live in the Netherlands, then check out my channel over here where I upload travel vlogs. So I'm going to get on with my dinner now and then get some studying done. So until next time.


Thank you. I am super excited to be here today to talk about Delta Lake and specifically how it can help bring data reliability to your existing data lakes. But before I get into that, I want to talk a little bit about data lakes and why everyone's so excited about them. Because I've heard lots of stories of people spending millions of millions of dollars on their data lake project and sometimes it doesn't want to work really work out and I want to understand what that means. But first of all, what is the promise of the data lake? And as far as I understand it, it's something like this. Your organization has a lot of different types of data. It could be customer data, it could be metrics, it could be console logs, it could be unstructured things like images and video, and the idea of a data lake is rather than spending a bunch of time doing ETL, coming up with a really strict schema and loading it into a traditional database. Instead, you're just going to store it all. A data lake, it's just a directory in a file system and you can just put it there. You don't even have to know if it's valuable ahead of time. And I think that's actually a good thing, because you often don't know ahead of time what data is going to be valuable and the ability to just collect it all and figure it out later is actually pretty powerful. And then the idea is, after we've collected all of this data, we're going to do some really cool things with it. We'll do data science and machine learning, we'll build predictive models, we'll maybe even cure cancer with genomics. But I have some bad news for you. In most cases, your data is garbage. And the reason for that is, is varied. It could be that someone upstream from you decided to change the format of a date without telling you about it. It could be that some machine is dropping records and they're showing up days late now. It could be that you're missing, there's two different data sets and you need to join them together to be able to understand them. And the problem with this is, if you have garbage in, you're going to store garbage.  And so I want to talk about a pattern that I've seen play out over and over again in these data lake projects, where people try to build a lot of architecture around them to kind of solve some of these shortcomings. So let's start with a pretty typical problem here. This is, I bet, something that many of you have built. Your boss comes to you and you have a problem like this. You have events coming into some system like Kafka, but it could also be Kinesis or S3 or Azure Data Lake, and you want to do two things. You want to do streaming analytics, so you can understand what's happening in your business right now at this very moment. And you also want to do AI and reporting, where you take a longitudinal view and you actually look at historical data and you build trends and predictive models for the future. So I'm a little biased, but if I was given this problem, I would start by using Apache Spark. So we have, you know, great APIs that connect to Kafka. We support event time aggregation. You can write your code in Scala, Java, Python, or even just pure SQL, and you can do streaming analytics from that. Great, right? We're done. Unfortunately, this brings us to challenge number one, which is historical queries. Systems like Kafka are great for storing a day, a week, maybe a month of data, but they're not great for storing years and years worth of data. So I spent a lot of time looking around on the internet, reading blog posts, and I came up to this thing called the Lambda Architecture, which, as far as I can tell, means you're just going to do everything twice. We'll have one system for streaming and one system for batch, and, you know, Spark has unified APIs for streaming in batch, so it works out okay. And now we can collect all of the historical data in our data lake and from there, so we'll add that, you know, one extra complication to architecture, but it allows us to do both of these problems. And now with Spark, we can do AI and reporting. All good, right? Unfortunately, this brings us to challenge number two, which, as I said before, your data is messy. Some columns are null and you don't understand why. And so what a pattern I see is people will start writing extra scripts that do validations. So this is a whole nother set of Spark programs that are just checking the quality of your data and they're emailing somebody if something goes wrong. And again, because we have the Lambda Architecture, we have to do this both for streaming and batch, so it's another two things that we need to worry about, but we can add this to our architecture and now we've got validation, so we'll know when our data is messy. But unfortunately, some of that messy data made it into your data lake. There are already mistakes and failures corrupting those files. And this isn't a traditional database, so if I want to correct those, it's actually very difficult. I have to go and find them and fix them. I have to be very careful not to crash in the middle, or I might corrupt the data in my data lake. And what that's going to give us, that's okay. Okay, now I can continue. So what do you do to handle this? Well, a pattern that I've seen over and over again, and I bet something that many of you in the audience have actually done before, is you use partitioning. Rather than store everything in one gigantic directory, we're going to break it up into days, months, hours, minutes, whatever makes sense for your business. And now when something goes wrong, rather than recompute everything from scratch, I'll just build an engine that can blow away one of those directories, and I'll just recompute one day. And so now you can correct those mistakes and failures in my data lake, and we're good to go. But again, this is one more complication we had to add to our architecture in order to enable that. And then finally, this brings us to challenge number four, updates.  Or you have problems like change data capture, where you're constantly getting a feed of updates, and you want your data lake to reflect what's going on in reality. So again, we're good engineers, we know how to solve these problems. We'll write a whole nother set of Spark programs that can handle doing updates and merges. You've got to be really careful here, though, because if you modify data while people are querying it, they're going to get the wrong results. So we'll schedule this, so we'll only run GDPR at one in the morning, and we'll make sure that our batch jobs don't start until three in the morning, and we'll hope the GDPR job definitely finishes before then. But again, this is something that we can handle, and we add these updates. But what you'll notice is this is a very complicated architecture, and I think the theme here is you're wasting a lot of time and money solving these well-known systems problems rather than doing what you really want to do, which is extract value from your data. And the things that are distracting you from getting value out of your data are, I think these are kind of a good summary. You don't have atomicity. Atomicity is this property that when something happens, it either happens completely or not at all. And you don't have this in a distributed system and on a file system. When something goes wrong in the middle of your Spark job, you have to worry about, did it write something out already? Do I have partial results out there? Have I duplicated my data? And that can make it very difficult. You have to do a lot of manual cleanup when something goes wrong. There's no quality enforcement. It's just a directory out on a file system. So if anybody can drop anything they want in there, it's very hard to reason about what you're going to get when you query it. And finally, there's also no consistency or isolation. As I mentioned before, you have to be very careful not to schedule batch jobs over a table that's being modified. And similarly, it's almost impossible to make streaming and batch on the same data set. And so this is why we created Delta Lake. And the idea of Delta Lake is we're going to take this relatively complicated architecture where you're spending all of your time thinking about systems problems and change it to something that looks more like this, where instead you're thinking about data flow. And the core enabling technology here is this well-known idea called acid transactions. Acid transactions mean that it's atomic, consistent, isolated, and durable. So you can focus on your data flow rather than worrying about failures. If something goes wrong, it automatically rolls back completely and you don't have to do any extra cleanup. And multiple people can modify the same data at the same time and it'll be as though they're doing it one at a time. And of course, if I was going to be collecting petabytes of data that was very valuable to my organization, I wouldn't want to lock this into some vendor-specific black box proprietary format. And that's why Delta Lake is based on open standard and is fully open source. The data is stored in Apache Parquet and the transaction protocol for Delta is also licensed under the Apache license. And we actually just recently, a couple months ago, announced that we've donated the project to the Linux Foundation and formed a new foundation under the Linux Foundation as the permanent vendor-neutral home for Delta. And we have a growing community, including Spark and Presto and others, where we're trying to make it possible for everybody to read from the Delta Lake. However, today Delta is deeply powered by Apache Spark and so if you have existing Spark programs, it's almost trivial to convert them and I'll actually show you how in a couple of slides. Delta plugs into both the streaming and batch APIs of Spark, so your existing programs can run with confidence. So I want to simplify this picture now and talk about some of the patterns that emerge once you stop thinking about systems problems and start thinking about data quality. I'm going to talk about these different data quality levels. Now, these are not specific features of Delta. There is nothing that says you have to have a bronze, silver, and gold table or that you have to have at least one of them, but I think these are really useful vocabulary that you can use in your organization to talk about quality. So starting at the B.O. and the idea here is rather than just go directly from raw data to the finished product, you're actually going to incrementally improve the quality of your data and I actually think that's valuable and I'll talk about why. So starting at the beginning, we have our bronze raw data. This is just whatever is coming out of Kafka, Kinesis, S3, people will have raw JSON stored in a column and you might say, wait a second, why am I storing raw data? Why am I not starting by doing ETL? And one of the reasons here is there are no bugs in a parser that you don't write. If you keep the raw data, you can always go back to it and process from the beginning of time. And Delta is designed to store years and years worth of data. We had one of our customers who was storing data in another database and they were only able to store two weeks. With Delta, they're now keeping three years worth of data. And so after you've collected all of this data in bronze, the next step is to move it to silver. In silver, you start to do some of the cleaning. You maybe filter out records that you aren't interested in. You take that JSON and you parse it into top-level columns. You do joins to augment data so that you have extra information that's useful in that processing. And you might say, wait a second, why am I materializing this intermediate data if it's not my final answer? And there's actually a couple of reasons for that. I think the most obvious one is maybe this clean data is useful to multiple people in your organization. And by creating this silver table, multiple people can use this in their final answer. So maybe you've created a really interesting, featureized data set and multiple people can train their models on it. But something that actually surprised me that a lot of users came and said was very valuable is silver tables are also useful for debugging. When something goes wrong in your final analysis, you actually have this table that you can query with the full power of SQL. You can ask questions like how many nulls do exist in this column or how many distinct values are here and things will jump out in you and you'll understand where the quality problems are coming from. And then finally, we move on to gold. This is high-level business aggregates, machine learning models, things that actually solve a business problem that means something to somebody in your company. And you can share this and do streaming analytics or AI and reporting on it. You can read it with Spark or Presto and a growing community of tools. Another pattern is people often use streaming to move their data through their Delta Lake. And you might say, wait a second, I don't need streaming. I don't have low latency requirements. I'm fine with batch jobs. And I actually think that's the wrong way to look at it. To me, streaming can do low latency and that's one thing that it's very powerful for. But what streaming is really about is it's about incremental computation on an ever-growing set of data. And pretty much everybody has a set of interesting queries that they want to run and pretty much everybody's data is constantly changing. The kinds of problems that streaming solves are the same problems you have to solve today by hand when you're running a batch job. You need to ask what data is new and what's already been processed. How do I take that new data, process it exactly once with no duplicates and no drop data and commit it downstream transactionally? How do I checkpoint where I'm at so I can recover if the system crashes? And these are all handled by the streaming system. And in Apache Spark, we actually have these really nice knobs that you can tune to trade-off cost for latency depending on your particular application. If you have a hyper-low latency application where you care about milliseconds latency, you can use continuous processing mode. We hold on to a core. We're continually streaming data through. There's no batching. And so it runs as fast as possible. But it's very expensive. Only your stream can use that core and nobody else can. In the middle, we have micro-batching. You can run many streams on a single cluster and each stream will run in tiny little increments multiplexing the resources of the cluster. Here, you can get seconds to minutes latency but run lots of streams on the same hardware. And then at the far end, many people actually have data sets that only change once a day or once a week or once a month. And in these cases, you can use trigger once. With trigger once, the stream starts up at the beginning, processes everything that's there, and then shuts down. And if you're using the power of the cloud, this is great because you can take advantage of elasticity. When your stream is done, shut that cluster off and stop paying for it. So you can save orders of magnitudes and processing costs by taking advantage to this if you don't have strict latency requirements. And so what you'll really see is streaming allows you to focus only on data flow rather than all of this control flow of processing. It can dramatically simplify the work. However, you know, batch jobs are also very important and Delta has full support for batch operations as well. When you want to do GDPR once every 30 days to remove all of those people who have asked to be forgotten, you can do that with an update or with a delete. If you want to do change data capture once a month, we have the merge into command where you can do upserts. If you want to every month delete data that's more than three months old, that's also supported. We support all of the standard DML from a traditional database, update, delete and merge. And you can also do crazy things like atomically overwrite multiple partitions. So pretty much any APIs in Spark can be used with Delta. There's one final pattern that I want to talk about, which is reprocessing. If you follow these patterns, if you keep all of your raw data in bronze and you use streaming to do processing, it now becomes very easy to solve cases where you might have made a mistake or where you come up with some new thing that you can't compute from scratch. And the reason for this is, when you start a stream with a fresh checkpoint, it's going to compute the same answer as a batch job over the same data set. The way streams work is when they first start, they take a snapshot of the Delta table at the moment the stream begins. They break that snapshot into a bunch of individual pieces and process them incrementally. When they get to the end of that snapshot, they start tailing the transaction log to only process new data that has arrived since the beginning. They can take advantage of the elasticity of the cloud. When you do that initial backfill, start up a thousand node cluster and process it really quickly. When you get to the incremental processing at the end, scale it down and keep 10 machines running so you can reduce costs significantly. But having that raw data means you can always go back to it and ask those interesting questions as though you had thought of them from the beginning and as though your code had no bugs in it. Now that I've talked about what Delta is good at, I want to talk about some actual use cases for using it. This is a relatively recent open source project. We just open sourced it in April. But it's actually been a product inside of Databricks for a couple of years. So it's actually used by thousands of organizations around the world. And last month, we processed over two exabytes of data alone. I want to talk about one particular use case that I think is really cool. This is from Comcast, which is a large cable provider in the United States. And basically what they have is they have data on every time anybody clicks their remote for any Comcast subscriber. And they really want to understand what people's content journey is like. They want to understand you're watching ESPN and then you go to the home shopping network, then you go back to ESPN, and they want to understand what that looks like. So they have this segmentation job where they're actually watching people through time. And as you can imagine, they have a lot of customers. And so this would actually max out their spark clusters. So they did what any good engineer would do when you have a scaling problem. They said, well, let's just hash partition it. They had one spark cluster. They had 10 spark clusters. And they would take all of their customers and they would take their user ID, mod it by 10, and spread them out across all of these clusters. And great, now it runs. However, now they have 10 clusters to manage, 10 schedules to work with, 10 sets of logs to comb through, and 10 sets of errors to deal with. It's a lot of extra maintenance. So when they took this and they converted it to Delta, because if Delta's more efficient metadata handling and scheduling, they would be able to manage the spark cluster. And they were also able to reduce the number of machines that they needed to use by 10x. So that's a dramatic savings. Not only in engineering time, but also in dollars spent to Amazon or AWS, or Azure. So that's pretty cool. So if you want to figure out how to use Delta Lake, it's actually really simple if you're already using Apache Spark. With this one command, dash, dash packages, you can add Delta to any existing spark cluster. And you can install it on the cluster. If you're building spark applications locally, you can add these Maven coordinates and it'll automatically download Delta and include it in your project. And then in your code, it's actually trivial. You just take the current format that you're using, whether it's Parquet, JSON, CSV, ORC, whatever, and switch it to Delta. And everything will work the same except now you'll have these power of ACID transactions and scalable metadata. I want to talk about something that's coming in the future, just to kind of let you know where we're going. This is a new project that we're currently prototyping at Databricks, and we hope to open source in the next three to six months. And the idea here is once people start using Delta, they go from having one stream and one Delta table to 10 streams, 100 streams, 1,000 streams, and it becomes very complicated to reason about what's going on. And so the idea of declarative pipelines is rather than managing each table as its own individual entity, we want to give you a language where you can express your data flow as one holistic graph. So data comes in at the beginning, and you move it through the bronze, silver, gold quality levels, using the same APIs that you already know and love. So in this example, I'm creating a new data set called warehouse, and I'm specifying it as this declarative query. For those of you familiar with materialized views, this is materialized views for Spark. The idea here is it depends on some other input, which is data coming from Kafka, and I'm going to use standard data frame transformations on it. We plan to provide these APIs in Java, Scala, Python, and also just pure SQL. In addition to the query, you can specify a lot of details about how it's going to be stored. So you can say where you want it to be located, if you have regulatory restrictions. You can specify whether or not you want strict schema checking. Delta actually has two different modes here. You can automatically upgrade the schema, so columns are added automatically anytime something new arrives. Delta can handle that, and maybe you want to do that in your bronze tables. When you get to gold though, maybe you want to be a little bit stricter, and you only want the schema to change when you plan on it, and so you can use standard DDL, Alt or Table, Add Column to modify it. Delta supports both of these modes. Finally, you might want to register it in a metastore, so it's discoverable by other people in your organization, or you might want to add a human-readable description that says where this data is coming from, how it was processed, what it's used for, so that other people can understand and then finally, my favorite feature here is this thing called expectations. Expectations allow you to take your domain knowledge about what quality means and tell the system about them. So in this case, I expect that this table always has a valid timestamp. And what is a valid timestamp? It doesn't just mean that there's a date there. It actually, for me, I know that Databricks started in 2012, and so if there's a record from, say, 1970, that's probably a parsing mistake, even though it's actually a valid date. You can actually express this as a predicate. You can say it needs to be after January 1st, 2012, and this sounds very similar to a database invariant. In an invariant, it'll fail any transaction where the invariant is violated, and you can do that, too. If this is a table that you're going to be reporting to regulators, you probably don't ever want any weird data to get into it. But earlier on in your bronze and silver tables, if you stop everything every time something unexpected happens, you probably won't get anything done. And so what expectations do is they allow you to tune the severity of an unexpected duple. So I can say, for example, if more than 5% of the records are invalid, send me an email. Or my favorite feature here is this notion of data quarantining. So with a data quarantine, when something unexpected is seen, rather than stop processing, we'll allow processing to continue, and we'll just take that unexpected record and divert it, route it to another table, where an engineer can come in and look at it at their convenience to understand what went wrong. And we don't just give you the bad duple. We actually plug deeply into Apache Spark so we can give you more debugging information. We modify the query plan so we can tell you, this was the record that came in, this was the record that came out, here is the expectation that was violated, and here's the code that produced that bad record. It makes debugging significantly easier. I've actually been using this feature on some of my data sets inside of Databricks, and it's amazing the things you discover about your data when you just write down what you expect and you see when those expectations are violated. So finally, before I close, I want to talk a little bit about how Delta works under the covers. I want to get into the nitty-gritty technical details because acid transactions on top of a distributed system like Spark sound almost too good to be true, but it turns out there's a couple of simple tricks we can put together to make this possible. So starting at the beginning, Delta on disk looks exactly like your data lake today. It's just a directory, but in addition to the data files, we also store a transaction log. And inside of the transaction log, we keep track of different versions of the table as it changes through time. So you can see these JSON files here are version zero of the table and then version one of the table. And then alongside this, we have optional partition directories. I say optional because they're actually just there for debugging. The actual information about partitioning is stored in the transaction log itself. This allows us to do really cool things like on S3 if you want to have maximum performance. What you actually do is you pick random directory names to spread it out across their metadata tier. We've actually had cases where Delta overloaded S3 and we had to do this. And then finally data files, which are stored in standard parquet readable by many of the tools that you're already using. So what goes in each of those different table versions? Well, you can really think of this as a change log. What we're doing is we're recording what's different from the previous version. So different actions that can be inside of that change log are changed the metadata of the table. So for example, if I want to add a column, I would put a change metadata action. If I want to add data to the table, I would add a file. And along with that file, you can add optional statistics about the data. For example, min max values for all of the columns to tell you what's contained in this. That can allow you to do really cool tricks like data skipping. And then finally, remove file. Remove file will take data out of it. And the result now, you can take the transaction log and play it through time. And what you'll come up with at the end is the current metadata of the table and the set of files that are currently valid. So we're using a trick here called multi-version concurrency control. On disk, there are actually multiple versions of the table simultaneously existing. And it's the transaction log that says which files are valid at any particular moment in time. So now that we've got that, how do we get those nice acid letters? Let's start at the beginning with Adamicity. So we want to have this property that let's say, for example, I add two files in. And then these are small files. So I want to compact them into one bigger file. So I'm going to add another action. And it's very important that this happens atomically. If I only do the removes and forget to do the add, I've lost data. If I only do the add and I forget to do the removes, I've duplicated data. So it's very important this is atomic. So the trick we use is we'll actually use atomic primitives of the underlying file system. On S3, you can get Adamicity kind of automatically because the put command starts by saying how many bytes to expect in that put. And if it doesn't get all of those bytes, the put will not finish. It will automatically fail. On systems like HDFS or Azure Data Lake, we'll use transactional rename. So we create a temporary file and we rename it to its final destination, and they guarantee that that will be all or nothing. And so now we have this nice ordered atomic history of the table. So now for consistency, we need to all agree on the order that those changes happened. And so here, we need a property called mutual exclusion. So user one can create version zero, and that's great. User two can create version one, and we're all good. But if both users try to create version two at the same time, it's very important that one of them succeeds and the other one gets a message back that says, hey, that version already exists. So again, on S3, that's actually pretty difficult. S3 is specifically not a lock store. It will accept any number of rights to a file, and it will not tell you who won. And in fact, it takes sometimes hours to figure out who won. So we have a separate service that sits out on the side and mediates who's actually there. On systems like HDFS or Azure Data Lake, it's actually guaranteed by the file system. That rename operation we were talking about before will only succeed if that destination doesn't already exist. And so we get this nice property. But now, you might be saying, wait a second. If every time two people modify the table at the same time, the system just crashes, am I going to like waste a lot of time dealing with that? Well, fortunately, we have one more trick up our sleeve, which is we solve conflicts optimistically. There's this cool technique called optimistic concurrency control. And the idea is this. Let's say for example, I have two streams that are writing into the same table. They're going to follow this protocol to write into the table. They'll start by recording the start version. So they're saying, okay, as a version zero, I am writing into this table. They will record what they read and write from the table. So they're both going to read the schema to make sure that the data that they're writing in matches the schema that's expected. And then they're going to speculatively write out some parquet files, so they'll write those out. Now, they're not part of the table, even though they're there on disk until they've been added to the transaction log. They will each go and attempt to commit. In this case, user one wins, user two loses. And so what user two is going to do is they're actually going to go and look at what happened in version one of this table, and they're going to say, does this change anything that I care about? And in this case, no. The schema has not changed. The schema is the only thing I looked at. So I actually don't know if I happened before or after user one's operation. So the system will automatically retry and reorder these, so that nobody succeeds and doesn't even realize that something went wrong. The only time you'll have a conflict in delta is when two people are actually modifying the same data at the same time. And in that case, you do want it to fail because otherwise you might get the wrong answer. So you try again and everything's good. So one final trick here is massive metadata. When you have a very large data lake, you might have tens of thousands or even tens of millions of files in the data lake. One who's tried to put thousands of partitions in the hive meta store knows that this can be a problem. And so in delta, we want to support these use cases without you having to do a bunch of extra scaling work. And so the trick we're going to use here is we're going to take metadata and we're going to start thinking about it like data. And fortunately, we already have a big data processing system here that we can use.  A checkpoint is a snapshot of the table at a moment in time and you can read that rather than playing the log forward. And the really cool thing, since the checkpoint is stored in Parquet, is you can actually query it very efficiently with Spark. So let's say you have a table with 100 million files in it and you want to query just yesterday. Well, we'll actually do this in two parts. We'll start by using Spark to query that checkpoint to identify only the files that are relevant to your query and then we'll actually go back and query it. This trick in many cases can eliminate significant amounts of data from your query. In one of our customers' use cases, they're a large Fortune 10 company. They want to do intrusion detection over their network. They capture basically trillions of TCP connections per day into their Delta Lake. And what they want to be able to do is they want to be able to ask questions like, what did these two computers say to each other during this time? By using the checkpoint and by doing efficient data skipping and filtering, they're actually able to eliminate 97% of the data in the table before actually even reading it. And this was able to take their queries from running in hours to running in seconds, which is pretty powerful. So before I close, I want to talk just a little bit about the future of the project. This is a relatively recent open source project. Today, we don't have full parity between what's available in Databricks and what's available in open source, but my commitment is all APIs need to be open. And so my team has spent the last three months outsourcing everything as fast as they can. And I think by Spark 3.0, we should have full parity. So, you know, in the last couple months, we've added support for S3 and ADLS. We added the DML commands, update, delete, and merge. We added the ability to convert existing Parquet tables to Delta without reprocessing any of that data. We added support for Python. We added history, so you can kind of do linear audit tracking of how a Delta table has changed over time. And I'm really excited about the 0.50 release, which should be coming out in December. It adds improved concurrency, so, you know, multiple transactions that don't actually overlap will no longer conflict with each other. We had the ability to write out manifest in Presto's format, so you can query Delta tables with Presto directly. And finally, we're making the merge command even better when you're doing operations like deduplication. And with Spark 3.0, we're going to be plugging in to the data source V2 API. And the reason that's exciting is it'll allow you to do things like create table and alter table and register those tables in the hive meta store. So, I'd like to invite you to check out our website. It's just delta.io. It has a getting started guide, so you can check out, you know, how to download it, how to use it, examples for all of the cases I talked about here. There's also links to our Slack channel. My team spends all day hanging out in Slack answering technical questions about Delta. So, please check it out and join the community. Thank you very much.