Hello, welcome to the Jenkins online webinar. Today is August 6th, and we have a presentation by Olivier about how Jenkins builds and delivers Jenkins in the cloud. It's your first time at the Jenkins online meetup. Basically, it's a event organized by Jenkins contributors, where we focus on presenting different things about what happens in the project and in the community. It's new features, developer tools, also ongoing development and preview of things you may see soon as users. The main goal is to actually show something, talk, and we encourage you to participate in the discussion. So you can use Zoom Q&A. After the call, we will also have a chat, so we invite everyone to stay. And yeah, the goal, again, anything about Jenkins. We also run a series of meetups about Jenkins and Kubernetes. And actually, today's meetup is also about Jenkins and Kubernetes, though it's a bit specific because it's our own project's key study. So the release infrastructure Olivier will be presenting today around some Kubernetes. And there is a lot of various things to learn about. So we are prepared to such key study. In general, we're interested to have any talk about the integrations between Jenkins and Kubernetes, like plugins, features, health charts, tools like operator or integrations with other CNCF ecosystem projects. And if you want to present, let us know, because we're always looking for speakers. Regarding our current meetup schedule, so we have a meetup today. The next confirmed meetup is August 17th. It will be about GitHub apps, authentication, and checks API integration. We also plan to have JSOC project demos in the end of August. We may organize some additional meetups in the middle because we usually announce meetups with one week advance, and there might be more meetups in August. If you're interested to present, just let us know. If you want to do so, there is a page which describes how to do the process. We have an online open CFP process, so you can just prepare Google Docs, submit to the mailing list, and basically that's it. So speaking of the today's meetup, again, we encourage you to stay in the discussion and ask questions. So if you want to ask a question, please use Zoom Q&A. This is a button in your control panel. If you ask a question, you either answer it asynchronously or ask the speaker after the main part of the presentation. And again, you're welcome to stay until then, and then we will have open discussion. So basically, we will stop the recording and we can discuss any topics related to the presentation or to Jenkins after that. If you want to ask questions after the presentation, for example, if you're watching the video on YouTube, then please use chat, use information list, or use the feedback form which we will be sharing in the chat soon. OK, that's it from me. And again, thanks to everyone for participating in the meetup. And I would only be to do the main part of the presentation. Thanks, Alex. I'll share my screen now. I guess I can. Zoom is doing some interesting thing here. Interestingly enough, it doesn't allow me to share a window. It only allows me to share a screen. Oh, this stuff. So you're going to share your screen, right? I can't share the full screen, but I cannot share one specific window. It's unsupported and Zoom. OK, then let's do it in a different way. Sorry for that. I'm just reorganizing my window so I can just share a test of one. Can you see my screen? Yes, we can. So if there is anything wrong with the slides, feel free to tell me. So hi, hi, hi, everybody. Thanks for attending this session about how the Way to Jenkins project is releasing Jenkins. The Jenkins infrastructure officer for the Jenkins project and I did this project this evening. So the presentation is split into six parts. The first part I'm going to talk about what the Jenkins infrastructure project is. Then I'll clarify a few things and then we'll go into the detail more specifically about what the infrastructure looks like for the release environment, how we configure the release environment, what the process is to trigger a release, and then we'll conclude this presentation. So first, it's important for you if you're not necessarily familiar with what the Jenkins infrastructure project is.  So the Jenkins project, like any organization rely on infrastructure and provide services like the main website, the ticketing system, and a lot of different things. And so in this case, we are also working on the release. So the people contributing. So it's an open infrastructure project. So we really invite a lot of different contributors to participate. As you can see in those slides, people are spread on multiple time zones in multiple locations, which means that it forces us to put in place some practices, like having asynchronous communication and also if you rely on testing and stuff like that. There is no really rules about what it's in the Jenkins infrastructure project. But what I would say is there is an organization on GitHub called Jenkins infrastructure where we can find a lot of different repositories that describe multiple things. Like the main Jenkins.io website, the PEPET code that we use to manage infrastructure, a specific configuration for Kubernetes clusters that we manage and so on. So if you are looking for something in particular, that's usually the best place to go. And otherwise, feel free to just ask on RSC. You should find all the information on the Jenkins.io website. Before we continue, I would like just to clarify a few things. So those are not like real knowledge, but just to be sure that we talk about the same thing. And the first one is during the presentation, I'll be using the term Jenkins master. I know that we are changing this and we use a different one. The main thing is we don't have validated the results about using a different name. So I've been thinking about should we just use a different one? Should I use the one in the results? Whatever, and I just prefer for today to just stick to the Jenkins master name so we all understand what it means. Another thing is during the presentation, I'm going to talk about different Jenkins instances. And the Jenkins project managed a few of them for multiple reasons. So we have if we go from the left, we have trust.ci. So that Jenkins instance is running in a private network. Only available to few maintainers. And we use it to build the main Jenkins website, update centers, or any other job that we just want to be sure that only a group of people can access it. This machine is running on a virtual machine. Then we have cert.ci. This instance is used by the security team for security work. We probably didn't mention too much about that one. We then have infrared.ci, which is the one that I will talk during this presentation. It's mainly used to manage the Kubernetes cluster and to run some tasks for the infrastructure on a Kubernetes cluster. We have in the middle, ci.jnkings.io. I have ci.jnkings.io, which is the way we usually mention it in the main list. This one is the major and the biggest instance that we have. We use it to test the core. We use it to test plugins, library, whatever. This is probably the one that you already saw if you contributed somehow to the Jenkins project. And finally, the biggest, the one that I will talk mainly today is released at ci.jnkings.io. So this is the Jenkins instance used for the releases. While all the people, while it's quite, I mean, as soon as you have a service instance running in a VPN, so if you have access to the VPN, you can see the job results. But only a few people can trigger a job there. So both release.ci and info.ci are running and configured on the Kubernetes cluster. Another thing that I would like to clarify here is about the release date. So before I go into the different components, I just want to say that you may have seen previously that we now manage all the release from the new environments. So the previous date was we needed because, okay, so the founder of the project to trigger a new release, which obviously puts strong dependencies on him and his machines and his house. It was quite hard to contribute, even if the script used for the release were public, we had no access to the machine. So we couldn't help him to debug and to release, I mean, to really understand what's happening in his machines. And also it's why, I mean, at least a good thing is it was like security by obscurity because we had no access to the machine. We assumed that it was working correctly. And so basically we created a GERAT ticket on the R-ticketing system, so infra-910. And the idea was to do all the work needed to move from a machine running in a basement to release Jenkins to cloud services where Jenkins community had the control on the release environment and could trigger a new release if it was needed. So that's basically how the work here and it involved obviously not only configuring Jenkins, but providing the infrastructure, providing the service and the process to do the full release, which bring benefits like everybody can audit because we try to be as open as possible. Everybody can contribute to all the parts, but we wanted to be sure that only a few people could trigger release, but obviously because we moved from a machine running in the basement to the clouds, it was also harder to secure. So this is a really big overview of the different components and I will now go into the detail for each of them. Sorry. So now let's talk about, so now let's go a little bit about the infrastructure. So basically as everybody in the everywhere under this presentation, you are going to find the feelings. So each time I specify the Jenkins organization and the Git repository. So in this case, all the things related to the infrastructure is located in the Jenkins and fresh Azure repository. Basically we use Terraform to describe resources running on Azure and everything was designed to use Kubernetes back again. So basically, sorry. So for our infrastructure, we need few resources. As I said, everything is defined using Terraform code. So you may see Kubernetes cluster with Windows auto-scaling nodes, Linux auto-scaling nodes. We define network configuration. We have DNS, most of the things there is managed in Azure. We also rely on Azure Key Vault to store the code sending certificate that you need in the process. We use, we started to store the secrets that we used to decrypt our secrets and so on. So there's not really a rocket science there. It's mainly we need a resource, we describe it. Once we commit the code to the Git repository, it's tested on Cialogenkeys.io. So we have LinkedIn tests, validates and text. And then what we used to do is once the PR was validated, we merged it on Tresor.ci and it was deployed from there. The reality is today is that for cost reason, we had to stop building the PR, the infrastructure for that PR, but basically what we're doing is each time we created a new PR, we were deploying everything from scratch. And so we could test that we were able to upgrade from a point A to a point B or we were able to provision everything from scratch. And obviously it was costly. And then we also had to disable that job from Tresor.ci because we lost faith in our codes, which is, I mean, I'm sad to say that, but that's a good state. And basically I learned a few things, working on that part is first terraform testing, terraform is quite difficult. It asks you to constantly test your terraform codes. Terraform relies on third API that evolve, we deploy services that evolve on cloud vendors. So it means that we have to constantly test it to be sure that it still work. Testing your terraform code, the best way is to provision resources, but at the same time it's quite expensive. And finally, automating terraform apply can be risky because if you have states in your resources, you don't want to take the risk to delete accidentally a resource. So you want to each time review what are changing. And so that's one of the things that I learned working on this. Some resources could be automated and others not. And finally, there are tools that were created to solve these issues. So if I have to work on this and otherwise, if I still have the time, there are a few interesting tools that you should look after, which is ConfTest, a terra test of TFSEC. I only saw some demos there. I haven't had the time to really deep dive into those tools, but this is definitely something that we miss in order to enable the way we manage our infrastructure. Another thing is more specifically for the communities cluster, the AK and the AKS that we are using. Something that I learned more than like the hardware is you have a lot of feature that need to be configured at creation time. And so we really have to be sure in advance what you're going to need. And otherwise, you just end up by recreating the cluster again and again. So at least the good thing is you test your terraform codes, but on the other side, yeah, just not using the best way of time to test your infrastructure. Otherwise, again, as I said, it's not really rocket science, but if you are interested to really go more deeper in the terraform code here that we are using, everything is located on the GitHub history. So just usually the loop to contribute or to understand how to participate in the Jenkins Infra project is first by discussing. So we use Tira, Madingly, it's hard to see. The first step is first to identify if there is a need and if there is one, you're probably the best person to implement the change needed here. Once you did that implementation, you test that on CIDO Jenkins.io. If the test are green and someone can review the PR, then we merge the PR, we deploy, we validate it, and then we improve and the loop continue again and again and again. So the good thing is working on infrastructure. We usually have to work another first time and then you can just reuse the work that you did. Then once you have your infrastructure in place, you have to configure and deploy services running in there. So either you rely on cloud services or you maintain yourself. In our case, we have both because we are using communities, we even rely on the current and how to deploy our application. And obviously we have Jenkins and a few other services. So if you really need to participate to the release environment, one of the first thing that we had to deploy was a VPN because Jenkins didn't have the VPN. And if we wanted to move from a machine running to a basement, we had to be sure that only a group of people were able to access our services, our trusted services, sorry. We also use HeldUp to define group and manage access. We have Jenkins instance called release.ci. We publish our artifacts to our Maven repository called repo.jnk.jnc.org. And finally, all packages can be downloaded from package.jnk.io. Those are public services except it's not all except that release.ci, but still. And we have two main, three main key repository involved here. So we have the first one, which is about open VPN that lets you, that allows us to configure the open VPN Docker image, but also to configure it and to specify who can access the VPN access. So we both use HeldUp authentication for the VPN and certificates. So someone need to trust, to assign your certificates. So you could connect on the VPN. We have the Jenkins infrastructure chart repository where we define all the stuff related to our communities cluster, how we configure Jenkins, how we configure Nexus for some, I mean, for local Maven repository cache, how we test, I mean, we have, yeah, we have a bunch of services running on the communities cluster. And we also use a puppet to manage the VPN machine. The main reason is because the VPN machine need to have access to different networks, private networks. And I wanted to be sure that we had network interface in different location, but it was just easier to do it on a virtual machine. If, because I think at least in my case, I'm more interested to share about the communities and how we manage this. I would just highlight the different tools used to deploy and configure release that's the ID of Jenkins.io. So we mainly use Helm as the package manager for communities. So communities is just a way. So if you're not familiar with that, it's just a way to describe resources. And so you just say, okay, you need those resources and communities will convert that to real resources. So you just provide them with files and communities will create the volume, the network interface, all the things that it needs for you to deploy your application. And a Helm package is just a way to describe all the resources for a specific application like you have a Helm package for Jenkins. Then you have Helm file. Helm file is just a way, just a tool that allows you to deploy and manage your Helm application on your cluster. So you just run a Helm file apply and then it will ensure that the right configuration is used with the secrets and so on. We use SOAPS. SOAPS is a small tool to encrypt and decrypt the secrets and I'll talk a little bit about that in a few slides. And then we use another tool that I wrote which is the CLI to automatically update files based on some strategies. So when you look at the deployment, so the Jenkins and Frasier charts, you have a lot of different directories there. So I will not show you a specific configuration because I think that if you are interested by that specific session, it's just better for you to go on that kitchen repository and look and just learn the way we are doing this because we have a lot of codes, but just give you some guidelines about what to find where. So the short directory contains custom charts or custom charts. The Helm file directory contains Helm file configuration files. So basically we have one per application and that's where we define. We need that time application deploy in that namespace with that configuration using those secrets, basically. The cluster directory contains the definition of our main cluster. At the moment when we have one, I mean it depends sometimes we have to, we probably deploy more cluster, but it depends if we have the time to work on that. But for now at least we have one. We define configuration secrets. We have a Jenkins file that work on communities. So we just define the different stages that we apply when we want to deploy our application. We have the pod templates. Also I'll talk a little bit about that in few slides as well, but just to give you a big overview, the pod templates define the environment where you are going to render Jenkins file. So if you need a container with Helm, that's a big place where you define that. And finally the update CLI is the directory where you specify all the strategies that we want to apply to update files. I'll talk that more specifically in one slide. So if we go back to the secrets, we are using SOAPS. SOAPS is a tool from Mozilla. It allows you to store your secrets in a Git repository, which makes it quite easy to audit. You can easily roll back a version. So you can configure secrets, roll back to the previous version and so on. You don't have to maintain a third service. You can have different keys per file. So I find it's really useful in the case of the Jenkins project where different people have access to different services. And so I can easily delegate the permission, the management of, let's say the release environment to a sub set of people while other maintainers have access to the application. And finally, we can download multiple ways to decrypt and re-encrypt the secrets there. So in the case of the release project, we have an Azure, we use Azure Key Vaults to decrypt those files. And the Azure Key Vault is inaccessible from a specific private network. We use the GPG key for our contributors. So if you are granted access to that Git repository, we probably basically need a GPG key. At least in the case of the secrets, it has been working quite well. We try to not put everything in Git repository. So on purpose, we decided to just put the code-signing certificate elsewhere and the same for the private GPG key. So those are stored on Azure Key Vaults with limited access, but then we need the Git repository to access those. So yeah, here it's just to not put everything in one location. Another tool that we are using, which is something that I'm using to apply strategies. So in this case, what we have is, we just define a source. What's the rule to update a file? So in this case, we just say we want to use, we want to specify the Docker digest. So each time the Docker, so we fetch on the Docker registry for the image check-in slash check-ins kind of a test dash to the K11. And we fetch the digest. And then we check with the target. So the file located here, we check inside the key Jenkins.master.image tag. And if the digest does not match the one that we retrieve, we just change the file, commit the change, open a PR, and so someone can review the change and just we move forward. There are different ways. I mean, we are supporting different sources. We can fetch version for the Docker digest. We can fetch version from GitHub releases. I mean, that's, yeah, it's just a small CLI. So just, yeah, define different, different condition. And so the second example in this case is when we want to update the weekly releases, we just fetch a version from the Mavin repository. We fetch a version. We test that there is Docker image that exists with that specific version. And if it's the case, we just update the target. So yeah, that's what we use to continuously either update the workshop version, the pod templates, the version, the image use the pod templates or any other YAML configuration. So just, this is what the job looks like when we deploy and configure the communities cluster. So it's trigger every 30 minutes. And so we need the secrets. We just check if we can update the files. If we can, we just update them. And so we do the GitHub and some tests on the files. And then we just run the ham file apply. And so if it's needed ham file, we'll ensure that the activities cluster has the latest version. And these run every 30 minutes and it's been working quite good for us. Something specific to our environment is the way we manage the Jenkins master. In this case, it involves three things. So obviously, ham charts, plugins and the gcast configuration. For the ham charts, we made custom Jenkins charts, but basically most of we just have a strong requirement on the upstream one. So we just try to reuse as much as possible from the stable slash Jenkins ham charts, but we just override default values for our environment. And we also add a specific communities resources in our case. We have a third ham chart called ham dash Jenkins dash release. But it's getting less and less important over the time because the way to configure Jenkins on communities evolve. And so this is something that we try to do as well. So now most of the configuration is done inside the gcast. So that's why that ham chart is becoming less important. So the two most important plugin in our configuration is obviously the communities plugin because it allows us to run our load on communities and the configuration as code, which is a central piece of our infrastructure and really allow more contributor to really understand what you are doing, how we are doing it. And if someone identifies something that can be improved, they just open a PR and it will be applied. Next time our job is running. So inside the gcast you can find the cloud configuration for communities. In our case, we specifically override the default GNP pods because we have Linux and Windows containers. And we want to be sure that those are running on specific nodes. So we just specify node affinity that need to be used for every container in our cluster. And that's it. We have the configuration, the credentials for the Jenkins master. And then we define the job this. We judge the job using job this. So we have on the release environments mainly, the core and the remote team components. I'll give you a demo at the end of the presentation if I have the time. Regarding the way we manage the Jenkins agents, something important to understand when you use Jenkins agents on communities is that the environment is defined using a bot template. So a bot template on Kubernetes, it's a resource where you just say, my bot templates contain those containers. You can have one or you can have several containers. The way the Jenkins agents work, you always have the GDNP container which establish the connection between the agent and the master. And then you had as many containers as you need, depending on what you want to use in your Jenkins file. So you have different ways to specify that bot template. So either it's defined inside the global Jenkins configuration. Personally, I prefer to not put any configuration there except it's the GDNP containers because if for some reason we have to configure the GDNP connector in a specific way for the release environment, like for the node affinity, it's defined at the global configuration. Otherwise, everything is defined either in a shell libraries and in the case of the release projects, everything is defined close to the Jenkins file. And close to the Jenkins file, you can either inject snippets of your YAML bot templates or you can reference a bot templates next to the Jenkins file which is definitely the one that I prefer personally because it allows you to run LinkedIn tests on your bot templates and also use any tools that manipulate YAML. So that's why you always see in the GDNP repository that we are running in communities, you always see a Jenkins file and a bot templates.YAML are multiple bot templates if it makes sense. Regarding containers, something important that you also have to keep in mind is most of the time if you use containers that you haven't built yourself, they will probably not use the user ID, the UID 1000. It's usually better to use the same UID than the Jenkins agents. It's also usually a good idea to override the default commands because we want to be sure that the container is not stopped after a few seconds. Like say, if you use a container for just the CLI inside, you will probably have to override the command so that you can sleep for, let's say 99 days or whatever you need there. And finally, I try, I mean, I really like because this is something that's easily to manage on communities, I really like to use persistent volumes when it makes sense. So again, it's really dependent on the situation but in the case of the release environment, we have a specific website that allows you to download that allows you to download the file from a location and that website use the same volume, a user volume that is also mounted in the CLI environment. So we just have to copy the move of file into the right directory and it's immediately available from to all the people that rely on that specific website. And finally, one of the most important thing to keep in mind when you work with Jenkins agent is to use Bud Limits. Bud Limits allows you to have auto scaling cluster but on the other side can be really frustrating because if the limit is too low, you are going to face really weird issues. And on the other side, if the limit is too high, your pod will never be scheduled on any nodes or we just create impression nodes again and again and again. So yeah, and as soon as you start using Bud Limits, it's just better because you're going to face interesting issues there. So on a global basis, if I can let communities manage a resource or instead of Jenkins, I just try to let communities manage it. So what I wish I knew before working on this song, just keep in mind that per system volume, if you let communities manage them for you, may not really be persistent because if the communities die or whatever, just lose the volume there. So if you really need persistency, then it's better to manage communities out of the volume outside communities, let's say using Terraform. Another really great tool that I'm a really big fan is Set Manager because Set Manager allows you to generate the SSR certificates. And when you work with a lot of private services that do not have public access, it's just a game changing because you don't have to connect in a service and just accept an SSR certificate. Another point important is regarding the Git storing your security in Git. Just be ready to rotate them. I mean, it happened, but just pushing your security in an encrypted way is just really the best way to lose a lot of time and it can be quite easy to do. So, and I mean, I heard mysteries about people who just accidentally pushed the secrets, not encrypted, not encrypted. So just be ready that if you put your secret in a Git repository, you are able to change the secret and rapidly because yeah, it happens. Personally, in my case, I remember one time I pushed all the secrets and then when I just reviewed the PR, I realized that maybe my files were not encrypted. And so while it took me a few minutes to change the secret, to verify that again, it was not possible to use. It took me days to verify that nobody were able to abuse of that. And also what I found interesting is that it's just a few minutes for third vendors to the date that I pushed the secret publicly. So I received a lot of emails from companies saying, you should use your tool, you shouldn't basically push GitHub token publicly. So I tried to put in place few things to mitigate that. So first now, secrets are not pushed to public repositories. So there is a private repository where only few people have access to that. And also I always try to double check that I don't have any secrets leaking somewhere. But yeah, still, it happened. But I mean, in my case, the good thing is nobody was able to have use of that. But yeah, just lose days on that. Community auto scaling feature is really powerful. But again, because for that you need pod limits, you're going to face really weird issues. And so sooner you start to work with pod limits and auto scaling, better it is because I remember in my case, I first work without the pod limits and then I enabled the pod limits and I just lost quite, I mean, I just lost days. Just understand why my job was running and not anymore and what was that wrong issues about that file that was not able to be written and so on. So yeah, sometimes it's really weird. And finally, to debug pod limit issues, I think some monitoring in place that allows you to really understand how you are using containers and what's happening inside your containers on the Jenkins master, but at the same time on the Jenkins agents and easily correlate those information together, make it really easy to realize that you have a pod limit issue there and it's not something else. And in my case, yeah, primitive graphonite it was really easy to deploy. So yeah, good surprise during this process. How to contribute to this part again, you have few links across the slides. You have the Jenkins and from proprietary repository. You have the Jenkins and for a charge repository for the community's code. You have open VPN. And yeah, you should be able to find all the information by yourself if you are more interested about this specifically. And now let's quickly look at what, yeah, I don't have a lot of time anymore but I'll try to speed up a little bit. Just look at the release process what it looked like today. In this case, we're involving a few more tools Bash Maven, Python, Makefile. Before we continue, I would like to clarify that the Jenkins project has multiple release types. We have the weekly releases. So those are generated based on the master bench on the Jenkins code repository. It's done every week and it's triggered by a cron job. We have a test releases. This one is more carefully selected and we tend to freeze the code but also the scripts used to generate the release is one happened once a month-ish but yeah, there is no strict rules there. And then we have the security weekly and the security tests. The main difference compared to the weekly test is in this case, it involves security work and most of the work is done in private until the last moment where we publish everything. So obviously it increases the complexity of the release environment for that because we need promotion mechanism. So the release process is split into two different jobs. The first one is the Jenkins release. So the job is divided into categories, the Jenkins release where we fetch the code and we published a result on the Maven repository. So basically we start here. So we have the release code is defined on the Jenkinsiflash release. We define the Jenkins file that defined the different stages, the pod templates, what the environment looks like to run the Jenkins file and a few scripts. Then we fetch the code from Jenkinsiflash Jenkins. We published the artifacts to the GitHub repository and to Maven. And then we trigger a second job which is Jenkinsiflash release. Then we trigger to Jenkinsiflash release and again we have specific Jenkins file for the packaging. We have specific button plates for Linux and Windows packaging, a bunch of scripts and then we fetch the packaging script from the Jenkins CR packaging and then we publish the results on package.jnkns.io. The scripting involved here are multiple and it evolved over time that I worked on it. So obviously I started working in the Jenkins file. Then realized that I had a lot of complexity there and it was not this easy to test that. So I started using make files. Make files is really, that's a tool that I really love because it helps you to abstract common complexity. So let's say instead of having really long command the curve build, there's a lot of parameters. You just run make build and it's easy to read and it's easy to understand that in your case make build means that you're building a Docker image but the same that happened with the Jenkins file after a while you start to have quite a lot of complexity in the make file which become quite hard to read. And so that was the moment where we started at working on the bash scripts. So that bash script became the central piece of the release environments. And so I could use the same script from my Jenkins file but also from my machine. So I could test exactly the same thing. And then the bash scripts had most of the complexity obviously but it also became the place where I defined all the environment variables and the default values. The main reason is because it can be easy to override the value of an environment variable. So I prefer to define everything here and there is a convention to not define variable elsewhere unless it's really needed. Then I started looking for specific scripts. So I wrote Python script when I had to manipulate YAML, JSON, XML APIs. But still I prefer to keep the Python script as small as possible and always call them from the bash scripts. And obviously we have the Maven release plugin for this reason. So the release workflow looks like this. We check out the Git repository that we want to build. We plan that. So we just visualize what we are going to do. So with Git repository we check out where we will push the artifacts. We retrieve the codes and we certificate from Azure Key Vault. We grab the same for the GPG key from Azure Key Vault. Then we run Maven release prepare. We commit from the Jenkins file. So we don't use the Maven release prepare to commit because we want to be able to use different Git repository than the one defined in the pub.xml for those who understand what the Maven release plugin does. We stage the release and then we verify the GPG signature and the code sending certificates. Again, everything is available on the Git repositories. And for the packaging workflow, it's kind of the same. We fetch the GPG key to code sending certificates. We download the Jenkins.war from the Maven repository that we find in the previous job. And then we build and publish packages for each distribution. If we need, we do promotion and then we force a synchronization of the warehouse. What I wish I knew before working on this part. First, Maven release plugin seems to be black magic to me in the past and still black magic to me. Second thing is I'm trying to always use the right tool. In my case, I try to not over complicate something specific. So I like to use bash, I like to use Python. I like to use Jenkins file in some cases. And I always try to evaluate if I'm still doing the right thing. So that's why I put complexity in the Jenkins file, then move that complexity to the main file, then realize that it was better to maintain that in a bash script and so on. Try to always define an environment variable location strategy. So basically try to define your environment valuable in one location. In my case, I decided that the source of truth would be the bash script. Otherwise it's easy to just override the value into multiple location and it can be quite hard to review how you're doing. So and it's really important to decide where you put those environment variables because it will affect all the scripts you use elsewhere. And finally try to keep in mind what your environment looks like when you work on scripting because it doesn't make sense to write a Python script if your environment will not have Python and on the other side, it may make sense to add Python environment to your site. But again, it really depends on the situation. What I propose to do here is to share different things in this case. I can share with you what the release environment looks like. So we have two directory, one for the components, one for the core. When we go into the core, I hope you can read a plan. So yep, I'm connected. I went into the core directory. So there we have two directories for stable and for the weekly. And otherwise we have the core packaging. So we have a generic job for the core packaging and the same for the release. So if we go to the weekly, basically that thing is just, so we can build from the master branch. Basically what it does, it just triggered the generic packaging job. By the generic release job and the generic package. And so if we go to sit here, we'll see that we trigger the branch on the number 28. And so from here, let's open the ocean because I prefer visualization there. You can see that we do the different stages. And so in the case of a security release, the idea is kind of the same, except that this time branch, in the case of security or LTS release because we create a branch where we freeze the parameters. So in this case for the LTS, we can just go there. We just triggered this. We just triggered the job.  So in this case, we want to build a stable release. We can provide a few parameters. And yeah, once we've pushed the button, the release is sent. And regarding, I'll just go back to my presentation. Yes. So just to conclude this presentation, it's only the beginning of the release. And now that the community has the control of the release process, it's our role to make sure that it's maintainable, that we can regularly update our environments. And we are still looking for people who can take the ownership on the different components. So we have to maintain the infrastructure and that are from codes used for the infrastructure. We have to be sure that the way we configure our services are reliable and people can contribute to that. And finally, we have to be sure that we control the process and how we do the release. Thank you. Thank you, Olivier. Again, if you have any questions, please ask them in ZoomConnay. Yeah, one question we received is, for scalability, is it possible to have Jenkins running compromise and also run build engines in the cloud? So it's not directly related, but yeah. So what's the question? So if you run Jenkins on premise. Yeah, I'm putting engines to the cloud. And Mark, wait here, I do it all the time. So I run on premise and use cloud agents because I don't wanna host all the infrastructure on my local environment. So yeah, it works great. Now. Yeah, it's... Yeah, so actually it would be interesting to take a look at delivery pipelines but from the code side. So if you could briefly present how it looks like the recent infrastructure pipelines and ecosystem from the GitHub, what of you? I think it would be a great addition. So I'm not sure to understand what you mean by the delivery. So under how we published on which delivery pipeline? How basically the same pipelines as you presented and how we deliver them to the instance? So just to be sure that I understand correctly. So for example, if you go to the Jenkins in fresh release. So if we go, for example, for the packaging job we have here. Yeah, on the screen sharing, by the way. Sorry, oh, no, sorry. Can you see my screen? So basically, for example, in the case of the packaging we define in the Jenkins file that we want to use the Kubernetes cluster. So obviously the communities can be running wherever you want. It's just that by default is easier if your Jenkins is running under the same Kubernetes cluster because by default it has access but otherwise you can provide the configuration that you want. And then you specify a pod templates. In this case, YAML file is a pod templates definition. And then inside your Jenkins file you can say for example, from time to time. So by default here we are on Linux but for example, we have a job where we need to be running on Windows. And in this case we just say use the pod template definition provided on package dash windows. And if we look at what that pod templates looks like here is another directory pod templates. And so for example, if you look at package Linux we just define which image we want to use what are the limits and some. Does that answer the question? Thank you. Okay. So since we don't have more questions in the queue I suggest that we just stop the recording and continue with the live discussion. And thanks again to all of you for the presentation. We shared the feedback link in the chat and again we will appreciate your feedback so that we can make our next meetups better and provide more relevant information. Next meetup you like to happen in two weeks and see you there and so I'll stop the recording.


And it amazes me that stat, and I've shared with you, I think in previous shows, New York has the largest growth of homeschooling anywhere in the country, New York City. I wonder if this issue were put on the ballot in every state, is there a single state that would say no to it? I bet not. I bet it would be overwhelmingly passed everywhere. It would be my guess. Well, the national poll indicates that 67% nationally approve education freedom.


Hey everyone, you're about to watch a video that's going to get thousands of views. Very cool, but your ad needs to be right here. Give me a call. ARC recently awarded nearly $1.5 million and grant money through their power initiative to SOAR. It's going to give us an opportunity to do a lot of cool entrepreneurship training and development on college campuses, post-secondary campuses across Eastern Kentucky. The money will be used to teach college students at multiple universities about entrepreneurship, expanding workforce training and creating new jobs in several industries. With this money, SOAR hopes to encourage residents to open businesses in Kentucky with the hope of retaining population numbers. The thesis behind this project was pretty simple. Can we hire some additional staff into some real targeted events and programming? Because we know college students here in Eastern Kentucky are just as talented just as gifted as college students as anywhere else. And we know that in other parts of the country, these are the kids that are starting businesses. These are the kids that are getting into some high-tech, high-growth opportunities. For more information, you can visit SOAR-KY.gov. For Mountain Top News, I'm Kelsey Fidell.


My name is Arnoldo Melo, I'm a senior principal software engineer at Red Hat, and I'm Brazilian, I work from home in the Brazilian Northeast, you can see a focal of my backyard. So I've been working with the Linux Karma since 1998, a long time ago, and during this time, lots changed in this community. People came and people went, and the number of developers grew, and there are places where Linux has been implied as well. I started working with networking, and doing changes to the TCP IP stack to make it use less resources and scale better. And nowadays, I maintain the Linux observability tools, which is something that I will talk a little bit more in this presentation, as it is a way for you to learn how the various projects work by inspecting its internals. The interesting part that I've learned all those years while working with others is that you interact with people from lots of places, and the way that they think and how they expect things to work is different across lots of places. We have to try and observe how other people interact when working in these communities, see how they interact, and see how they submit patches, see how the reaction happens, and how they react to that, and learn from reading those conversations. It's one of the great aspects of open source that everything is discussed in public, so new comments can come and see how features were implemented, but was the interaction of the developers, and from there I learned a lot from the process. Where to start? Well, you can be working for some company, and then somebody will tell you where to start, but if you are interested in getting to an open source community, the best advice I can give is try to find something on your computer or in your smartphone that you don't like, that you would like to work in some other way, something which is not working. So, after that, you should try to locate the source code in the usual places, Google, GitHub, and then you can look at the source code, and if you know about that specific language that is used to implement that project, then you can already start to find where that thing that you want to fix is implemented. Another advice is to use observability tools, such as profilers, tracers, and the buggers, so that you can see what are the main functions and what are the main sequences of events that lead to that function, and just by looking at it, and especially looking at it when what you think is broken is happening, may help you in pinpointing where to try to work on your things. And when doing that, you may find problems, all the problems, as no code is free of bugs, so consider trying to fix things as you learn about the project that will bring good will from the maintainers, from the developers involved in the project, for when you decide to submit your new feature or the fix you started this process with. How to submit your work? Projects have documented processes, you should look at the source code for them, and then you're going to find how to submit, where to send your patches, to whom, who should be on the CC list, and most projects by now have also a code of conduct that you should read and see how what's appropriate and how to deal with the other developers. And please try doing it in baby steps. The smaller the patch you send, the better. I mean, if there are multiple things to do to get to your fix or to your new feature, you should just try to write as small as possible patches that go on building and eventually get to your objective. Because this is easy to review by the maintainers and by the other developers, you just have to look at the patch which is small, and then you see, oh, this is okay, let me go to the next one. And it helps finding problems later. Using a process that's called by section, where you utilize tools to do a binary search. You know that the first version works, and now it doesn't work. So you go to the middle in terms of change sets, in terms of patches, and try there. And then you go on saying that this works, it doesn't work, and you get more quickly to what was the change, or this problem was introduced. But what if people didn't like my patch? Well, should read a response, try to clarify things, perhaps it was not clear. When you first submitted a patch and then with some clarification, it may become acceptable. You should affect problems. Sometimes you made a mistake, or it didn't consider some specific case. So you should read the problem that were reported and addressed on. And then it should resubmit. Usually you send a new patch set, a series of messages, and then on the cover letter, on the first message, you say that's the second version, and on the cover letter, you describe what were the changes that you made in relation to the first version that you submitted. It's all about making progress. Sometimes the maintainer will not process all the patches, accept all the patches that you sent, but then it can apply some of them and share it picking, so to say. Some more recommendations. You should discuss publicly. It's not interesting to send messages in private mode, just to one person. Sometimes the developer doesn't have time at that moment, and then somebody else on the mailing list may jump in and help you. And sometimes even have more information about that than the person you thought at first should be the one that would help you. Be patient. As I said, you sometimes people are busy with all this stuff, and then can reply as fast as you would like. Don't be afraid to ask questions. It's just try to formulate the questions as clearly as you can. This is an interesting process because sometimes all the people who are reading that conversation learn by seeing the answers to your questions. And also consider participating in conferences. With the pandemic, most of the conferences like this one is being done over the internet. And the good thing is that the previous conferences are available on YouTube and places like YouTube. So please consider as well looking for previous recordings for things that you have interest in. So with this, I pass to Kate. Hi. My name is Kate Karsha. I'm an associate manager at Red Hat on the core Kernel engineering team. I myself am new to open source, but my team is very active in the upstream Linux Kernel community. So although I'm not making direct code contributions, I learned a lot about open source by supporting a mix of engineers, some of whom are very experienced reviewers or maintainers like Arnaldo, as well as engineers who are new to the space and want to get more involved, more like myself. And as a manager of a team that is working in open source communities, and as a woman in tech who is passionate for diversity, equity, and inclusion, I'm always thinking about how can I help to make open source a more accessible place for everyone? A 2017 GitHub study found that 3% of developers in open source communities identify as female, 1% identify as non-binary, and the study didn't look at race or other diversity groups. There's a lot of work that we have to do here. And so today I'm here to share with you, whether you be a manager or an aspiring manager, or maybe an individual contributor that would like a window into the world of management, how you can help at through some strategies such as hiring, mentorships, and work assignments that helps to make open source a more open place for everybody. The first strategy that I want to cover is about hiring for potential. So let's think about what potential means. Potential means having the capacity to become or develop into something in the future. So hiring for potential means that we're hiring people for their capacity to develop the skills that they need to grow successfully in the role, rather than hiring for the experience that they may have in doing the work that's required by the role. By hiring for potential, we want in the pool of candidates that we consider for open positions. And this enables us to add new capabilities to our teams, and in turn, add new capabilities to the open source communities that our teams work in. Now you may be thinking, well, how could you hire somebody if you don't at all consider their experience? Hiring for potential doesn't mean that we don't consider someone's experience. It's just that rather than evaluating if they have experience of already doing the job or work required by the job, we're evaluating that their experience is indicative that they'd be able to come in and learn and develop the skills that they need to do the job. So one example of this is hiring for a specific programming language experience. In kernel development, we use C. So in our case, if we were hiring for potential, one way we could do that is instead of only looking at candidates that know the C programming language, we could instead consider candidates also that have an understanding of low-level programming concepts, such as control flow or data structures, or maybe comparable languages, such as C++ or Rust. And this enables us to open ourselves up to the possibility of bringing somebody onto the team who may have a different perspective or may have something unique to bring to the table. So in summary, hiring for potential enables us to bring more perspectives onto our teams and into the open source communities that our teams work in. It's about opening opportunities for people that may not have had that chance otherwise or may not have even known that that was an option to them. Now we'll say, of course, just because you plant a C, it doesn't mean the plant will grow. And so we have to give that plant water and we have to give that plant sunlight. So in this case, when we bring somebody on and we hire them for potential, that alone is not enough. We also have to come in and help them successfully realize that potential. And that's a good segue into the next two strategies that I'll discuss, mentorships. So something that's really important as a manager is supporting mentorships. And mentoring is a voluntary ongoing relationship typically between a more senior associate, the mentor, and a more junior associate looking for development opportunities, which is the mentor. And in these partnerships, the mentor will give advice and guidance and support to the mentor. And in turn, the mentor is able to learn things that they may not have otherwise had the chance to come across if they were just working on their own. Now mentoring is really important for newcomers. The mentoring is especially important for underrepresented groups in technology. So research actually shows that one factor, one success factor from minorities in corporate America is having a strong support network and mentoring. But even more so, this research went on to find that a truly effective mentoring is not just about not just about having technical guidance or given being given direction. It's really about having a quality relationship with trust, where the mentee feels safe to express fears and concern to the mentor. And so as managers, we actually play a very big role in terms of helping to cultivate these types of relationships and to support them. So I'm going to give some tips that that I sort of follow in terms of working with folks that are seeking to be mentees and then also folks that are going to be mentoring. So for newcomers that are looking for a mentor, one of the things that I encourage them to think about is finding someone that they feel like they can be vulnerable with. Going back to what Arnoldo had spoke about earlier and his idea of learning by trying. Well, of course, if you're going to be trying something, one of the best ways that you kind of figure out what works and what doesn't is to make mistakes. And so working with somebody as a mentor who you can feel comfortable with to make mistakes, that really gives you the space as a mentee to learn. So I really encourage folks that are seeking mentors to seek out people that they feel that they can be vulnerable with. For more experienced engineers that are mentors, one of the things that I encourage is that they lean into the importance of the role. Now, of course, it is an opportunity for them to solidify their own skillset because they have to have a very deep understanding to be able to explain these concepts. But also it is an opportunity for them to leverage their experience and their expertise to help forge pathways for new people to join open source communities. And then for both the mentor and the mentee, something I work with is making sure that folks have the amount of time that they need to really dedicate to these types of relationships. It is an investment of time on both sides. And so something that makes it successful from my perspective is making sure that people have the time to really invest into it. Moving on to the next strategy is thinking about helping newcomers to identify work that is going to help build their skills, but then also build their confidence. And this is actually something that I do in close collaboration with mentors. So a couple of tips that I have regarding this is one, for somebody who's just getting started, I tend to recommend not picking up work that's going to be in the critical path or is urgent or has a tough timeline. And the reason for that is because I like to give people the chance to really dig into concepts and build a strong foundation of understanding. And if there is a timeline or if there's stress attached to something, I feel that can kind of detract from that experience. That's the one idea. The next idea I have about making sure that we can help newcomers identify work assignments is that one of my work assignments that can be broken down into adjustable chunks of work. And the idea behind this is really helping people to break down work so that they can get feedback as they do it. And that feedback includes positive feedback. Positive feedback means that they accomplish something. Aggression is actually quite motivating as well. I find that this is especially important in the case that you're working with new college grads. So research actually shows that one of the biggest struggles that new college graduates have in adjusting to the industry is the fact that there's not a lot of feedback. In college, you take courses, you have a syllabus. That syllabus has deadlines and assignments and there's rubrics that those assignments are then evaluated against. In the industry and in open source, we have so much freedom. I have freedom to define what we do, how we do it, what success means. And that freedom is wonderful. We want to harness that freedom. But helping newcomers through that transition period where they're able to learn how to manage that is really important. And so having a mentor that is able to identify a project and help the mentee kind of break that down is certainly something that can use sort of the challenge and maybe missing some of that structure. And also it gives the mentee or the newcomer an idea of how to break down work. So in the future, if they independently pick up more ambiguous tasks, then they're able to do something similar on their own. So one thing that I recommend in this particular instance is that for all of the project work, it's good to think about what is your definition of done. But definition of done doesn't always have to be a piece of code. You may break down a project and maybe perhaps some of it is investigation. And as part of that investigation, your definition of done is having a documentation that explains what your findings were. So going back a little bit to what Arnoldo said again, his approach about progressing, taking baby steps, rinse, repeat, and grow. I think that this sort of strategy of breaking work down needs really nicely into that. But then also, I think that is a great strategy to use as a manager as well. Everybody you work with is different. Coming back to the plant metaphor, certain plants need more or less sunlight or water to grow. And if it's very sunny, you might need more water and vice versa. And what I'm trying to say is that we have to be adaptable as managers to what people need and how people learn best. And that might look different for everybody. So if a project isn't working, we can change it. And if the connection with the mentor is not there, we can change it. So in summary, managers too can help make open source a more open place by creating more opportunities for people to join open source communities, by helping to support mentorships that provide learning experiences in a support network, and by helping to identify meaningful work that helps to build confidence and skills. So closing out our presentation today, we come back to the heart of what we discussed. Open means having no enclosing or confined barrier accessible on all or nearly all sides. We believe that open source is healthiest when different ideals and perspectives are included, welcome, recognized, heard, respected. We believe that open source is healthiest when it's truly open. And so hopefully you're leaving this presentation with some ideas, whether you'd be an individual contributor or a manager, and how you can also help unlock all that open has to offer. Thanks for joining us today. Well, thanks everybody for coming today. I know we're running probably a little over schedule since we had some technical difficulties, but we really appreciate all the good conversation and thoughts.


The United Nations has voiced alarm at reports of sexual violence in detention and disproportionate use of force against protesters in Algeria, calling for investigations into all alleged abuses. The UN Human Rights Office said it had received numerous reports of abuses in Algeria since weekly mass demonstrations, but the Iraq pro-democracy movement resumed in February. The Iraq protest movement was parked in February 2019 over President Abdul Aziz about a flicker's beat for a fifth term in office. The alien autocrat was forced to step down weeks later, but the Iraq has continued its demonstrations.


Αυτό είναι το φόρο, αν και ο πόνος που βλέπουμε αυτού του οικογένειον θα φτάσουμε την πόνο, όταν έχουμε φόρτα και οικογένειο, δεν το διδάξουμε, αλλά είναι ήδη the same as the previous, όπως η φόρτα, πρέπει να φύγει να be complete. Έτσι έχουμε δύσκολο του φόρου, όπως είχαμε για τις δυνειώσεις της εξοδεξής. Λοιπόν, σε κάποιο σκέφτος, το οποίο τα πράγματα κλάβουν, Είναι όταν έχουμε μεγαλύτερη χρησιμοποιή στην αμφιλή, που είναι η αγγέντρα, που είναι σε καταλαβαίνο της λογικής λογικής εξοδοσύνησης της πανδιφιλής. Αν υπάρχει από τη αμφιλή, στην αυτοσύνηση της πανδιφιλής, έχουμε αυτά τα φράγματα και το απελευθείο των αυτών πρόβλημας είναι πραγματικά σύνθετα με την αρκετά της πανδιφιλής. Λοιπόν, εγώ θα σας δώσει κάποια εξοδογή, γιατί αυτή είναι η κατάσταση της τελευταίας. Λοιπόν, έχουμε μιλήσει για το καλκουλούς και το αλζεβρα. Βλέπουμε ότι μπορεί να κάνουν όλα αυτά τα καλύτερα πράγματα, αλλά βλέπουμε επίσης και την ώρα, ότι υπάρχουν κάποιες πράγματα που μπορούν να κάνουν. Και η πιο στιγμή που έχουμε βλέπει στις, δεν ξέρω, τρεις δύσκολες πράγματα, είναι ότι η κλωσία της τρανσικής δεν είναι πρώτα ή δημιουργή, οπότε δεν είναι εξηγημένη στην αλζεβρα ή το καλκουλούς. Παρτικόν, ο αλζεβρα ή το καλκουλούς δεν μπορεί να εξηγήσει πράγματα που ανοιχθεί η εξηγημή, such as, yes. Of course. Oh yes, yes, yes. So, the aggregate operator, this is a very good question, the aggregate operator really involves second order logic in some sense, because what you do is you have an equivalence relation in action and you work with equivalence relations of this και υπάρχει πολλοί που θα κάνουν αγγραφίες, λιπκοί, στις άλλες, που έχουν κάνει τα δύο. Να φύγουν πολύ καλύτερα όταν κάνεις αυτό. Το άλλο που θα δώσω είναι πάνω από έναν τρόπο διαφορετικό λεξό. Όλη αυτή η δισκάσσυνή που είχαμε εδώ, απλήνει στις σημαντικές, ότι η εξογραφία μας είναι να γίνει στις στις στις. But in an actual database system, the queries don't retain sets, they retain multisets or bugs. In other words, when you do the projection, if you eliminate two columns, you may get identical tuples. The database system will retain these tuples, unless you explicitly ask it. This makes perfectly good sense precisely because you want to compute averages, aggregate operators. Let's say you want to compute the averages of all salaries. Some people may have the same salary in the same department, right? You don't want to throw this out, you'll mess up the average. So back in 1993, there was a paper by Surazit Saoud-Hourian-Mosevardi that they said, look, you database through a distance got it all wrong. You did Chandra Melling got it all wrong. You studied these problems with sets semantics. What happens with bugs semantics? Well, the situation is very embarrassing. We don't know if conjunctive query containment under bug semantics is decided. We don't know that. We know that conjunctive query containment with inequalities, not equal, is undecidable. That's a paper that Eric V, Jayram and myself had in post 2006, with a reduction from Hilbert's 10th problem. But the problem that's open is conjunctive query containment under bugs semantics. That's open. Is it decided? Ben here. Ben, you've tried this problem, right? With Swastika at some point, right? It's hard, right? Ben says it's hard, it's really hard. Okay. So yeah, there are a lot of things we still don't understand, including this one. So I want to finish by remaining, I have 35 minutes to tell you a little bit of the story of Datalog. So that's a declarative language that augments the language of conjunctive queries with recursion mechanism. So it's very amusing. In 1979, Eho and Ullman, you can't get more distinguished computer scientists than that, but they published a paper in POPL. This is the Programming Languages Conference, the top conference. And they showed that no relational algebra expression can define transitive closures. They didn't know that Fresen knew that back in 1954, right? And we saw this here with Erenforz-Fressen Games. But they discovered it and they had very good intuition. They essentially came up with one of the proofs that we saw here. They get credit for bringing to the Programming Language Community and the Database Community, in particular the fact that SQL cannot, as it was at the time, could not express recursive queries such as transitive closures. And now we understood from the discussion we had today, the last two days we understand the reason. Calculus, its first-order logic, can only express local properties. You can't tell apart using first-order formulas graphs of this type, cycles and unions of two cycles. In particular, this means if you have a database that has information about parent, you cannot write an algebra or a calculus expression that defines ancestor. Now it's interesting, of course, that if you think about paths in a graph or ancestor, this is an infinite union of conjunctive queries. But the result tells you it's not equivalent to any finite union. So this really suggests a severe limitation in the expressive power of algebra and calculus. So what could the people in databases do? Well, when you take the, when we teach the undergrads with the course, we tell the students that SQL is very nice, it's declarative, but if you run into trouble you can always bring in C or Java, right? Through embedded SQL you make a call and you write any recursive program you want. This is okay, but it's really a dirty solution. It's an inferior solution because the whole point of going to SQL and the higher-level language was to separate the design from the implementation. You go to a high-level and it destroys the high-level character of SQL. And the other possibility is to go back to the drawing board and see what can we do to augment the expressive power of calculus, of first order logic with some high-level declarative mechanism for recursion. In fact, this mechanism has been mentioned here under the name of fixed-point logics. I just want to give you a different perspective of this, but such a mechanism would be superior to the previous solution because it maintains the high-level character of calculus. So the dialogue here is the slogan. The dialogue is conjunctive queries plus recursion. It's what you get by forcing a marriage between conjunctive queries plus recursion and, in fact, the language was introduced by Sandra and Harrell in another paper, not the paper that Anous mentioned before in 1982, and since that time has been studied by the research community in great depth. I mean, there are literally hundreds of papers, scores of PhD thesis. It was when I entered the database field, I've been trained as a mathematical logician, but I went to my first database conference in 1987. I knew very little about databases, and I was like a child in a candy store. I could understand half the papers because they were about data log, and data log was familiar with because I had studied under the name of inductive definition. So by 1995, this had gone out of fashion in databases, and you found late 90s relatively few papers. In the last five or six years, there's been an amazing comeback of data log from all sorts of different areas of computer science. People have used versions of data log to specify network protocols. This is Joe Hellerstein and his students Bun Tao Lu and others. People at Microsoft have used data log as access control languages. There is a company in Oxford that is using data log in program analysis and so on and so forth. In fact, there was a conference on data log 2.0 at Oxford last year, and there was another one in Vienna in the next fall. So the language has found other applications outside databases, and finally the people that designed the SQL standard gave in and decided to introduce a version of data log that I'm going to tell you about called linear data log. So SQL 1999 and subsequent versions of the standard support data log in this restricted form that I will try to explain. So what is a data log program? It's basically a finite set of rules that express conjunctive queries. The only difference here is that before when we had a single conjunctive query, we had a name for the head that did not appear in the body. Here some names on the head may also appear in the body, and that's where recursion comes into the picture, and I'll show you lots of examples. So a data log program is a finite set of rules. It's expressing a conjunctive query, but some occur both on the left and the right. The ones that occur both on the left and the right are called intentional database predicates or recursive predicates. The ones that occur on the right but not on the left are extensional, and the idea is that the extensional ones are the given predicates. It's what we have in the database. The rules express some knowledge, and they are used to define the intentional predicates. That's what recursion is all about. So let's look at two examples. This is, again, the transitive closure reincarnated now as a data log program. So yesterday how to do it in least fixed point logic. Well, for us, here is going to be a rule, a data log program with two rules. So T says there is a path from X to Y. So how can we have a path of X to Y if there is an edge from X to Y, or there is a Z, and there is an edge from X to Z, and the path from Z to Y. Then there is the divide and conquer version, so to speak, in which you say you replace the second rule by TX, Z, and TZY, and that says there is a path from X to Y if there is a Z, such that there is a path from X to Z and the path from Z to Y. That's the divide and conquer. So intuitively, this is what these data log programs do. Of course, we have to give proper semantics and make sure that they, in fact, they define the transitive closure. We can allow, here is a program with several, with two recursive predicates. Odd says there is a path of odd length from X to Y, and even says there is a path of even length. So here one is used to define the other. You define there is a path of odd length if there is an edge, there is a Z, and there is an edge from X to Z, and the path of even length from Z to Y, and odd the other way around. So here we have two recursive predicates, two IDBs, even and odd, and one EDB. So we can think here that the data log program gives us a recursive specification of the IDB predicates, even and odd in terms of the EDB predicates is given to us, we use the data log program to define even and odd. So this is a case of mutual recursion, of course. What is the precise semantics of a data log program? Well, you can give two types of semantics, declarative semantics and procedural semantics. And then you can prove that the declarative semantics matches the procedural semantics. And that's what I want to do here. So the declarative, you can think of it as denotational semantics. You give an object, which is the meaning of the program, and the procedural or operational, you give an algorithm for computing these semantics. Now, I could do it through the least fixed point mechanism, but I want to give you a slightly different description of this, which somehow motivated from the first programming course in which we teach our students recursion. So when we teach recursion, we say, look at the factorial function, it has this nice recursive definition. And if I had plus and times, I can define exponentiation. In fact, there was a language called Pascal, I didn't have exponentiation explicitly.  So what's going on here? What is going on with this is that we can write recursive equations that define functions over the integers using plus and times. And then what we prove there, of course, in recursive function theory, is that there is only one function that satisfies this equation. We want to do something similar. We want to write recursive specifications. But what we want to do here in the recursive specifications, we don't have plus and times, all we have around are the operations of relational algebra. So you can think of a data log program as being a system of recursive equations where the operations are some of the operations of algebra. So here is how it goes. If I give you a data log program, this may have many predicates, many recursive predicates. For every recursive predicate, you write an equation. What do you do? For every equation, you take the expression on the right-hand side. That's a conjunctive query, so you can write it in algebra. And you combine the right-hand sides of the different rules for the same predicate using union. So, for instance, in the case of the transitive clause, we have only one recursive predicate, T. So this ST is E union. If I write the body of the second rule in algebra, it becomes pi14 sigma2 equal 3 T cos T. So that's a recursive equation that involves the data log program. And we can do the same thing if we have two predicates. Now we get a system of equations, one for every recursive predicate. So that's a system of recursive equations. Now, this is analogous to the situation of what we had with recursive functions, but unfortunately, it is not true that there is only one solution to these recursive equations. In fact, we can have many solutions. So in fact, every transitive relation containing E would satisfy these equations. So we have many solutions. So we can say that the semantics is the unique solution to this specification. So when we have many solutions, we hope to find a nice solution. And here there is the smallest beautiful approach. That will give you the least fixed point semantics. Or big is beautiful, you've got the greatest fixed point semantics. So it so happens that people chose here the least fixed point semantics. So the theorem is that every recursive equation arising from a datalog program has a smallest solution, smallest with respect to the partial order. If I had two, three, five recursive predicates is the extension of the partial order coordinate wise. So every recursive equation arising from a datalog program has a smallest solution. So in the case of this datalog program, the smallest solution actually is the transitive closure, right? So that's why if we take this datalog program, view them as recursive specifications and ask for the smallest solution, in that sense this datalog program gives us the transitive closure. This is the smallest solution to the recursive specification. And this is a very special case of a general theorem called the Kanaster-Tarski theorem which was not proved, it's not the joint paper. These are different papers. And interestingly enough, Tarski's paper that has this is his most cited paper, although it's one of his most trivial theorems. And it has to do with smallest solutions of recursive equations arising from monotone operations. So what was crucial here was that all the operators we have allowed in datalog programs are monotone. We have allowed disjunction, Cartesian product projection and in fact I'm going to just quickly sketch the proof of this. So we have these declarative semantics as the least fixed points, the least solutions to these specifications. Of course I haven't showed you that these solutions exist, but I will in a minute. Let's look at the procedural semantics. Well, we can give different meaning to the programs through a bottom-up evaluation. So what do we do? We have these rules. Remember, we have the given predicates and the intended predicates, the recursive ones. We start and we instantiate all recursive predicates to empty set. Now the right-hand side is a conjunctive query. So we can take this value and apply the rules, obtain new values for the heads, and therefore update the heads. Now with these new values of the head we plug them in and we repeat until there is no change of the IDP predicates. And when there is no change, we stop and we report this as the result of the data log problem. So this is the so-called bottom-up evaluation or naive evaluation where you have a four-loop or a Y-loop until no change appears. So for instance if you do this for the transitive closure P, you get a series of sequence of binary predicates. T0 is the empty set, Tn plus 1 is what you get by taking the rules and plugging for T what you had in the previous state. And so on. I'm not going to... These are all exams, I trust you can go over them. So here is the result that I want to get into. That if you have a data log program or true, the bottom-up evaluation of the procedural semantics terminates with a number of steps bounded by a polynomial in the size of the database instance and the declarative semantics coincides with the procedural semantics. And the proof is really not difficult at all. For simplicity let's assume we have just one recursive predicate and let's assume that the arity recursive predicate is K. So by induction we saw that the nth iteration is contained in the nth plus 1 iteration. This uses the monotonicity of the building blocks that we have. T0 is empty and then we use the monotonicity to get the T0 contained in T1 and we assume that Tn is contained in Tn plus 1 and then the monotonicity and the conjunctive queries gives us this. So we have an increasing sequence of binary relations but these are carry relations on the active domain of the database but this is a finite set. Therefore this sequence has to stop it cannot keep increasing. In fact there is a meme that most the size of the active domain to the K at which level we get Tm equal Tn plus 1 So before we hit this somewhere before up to the active domain of i to the K we are going to find that this iteration has stopped. So we know that this iteration has stopped at some point. So this was the termination and now we have found our solution Tn plus 1. So at this point we have a solution to the recursive specifications. Now we are going to prove this the smallest solution and we prove by induction that if we have another solution then every level of the iteration is contained in this other solution. Again, T0 is empty so it is contained in every solution and again we use the monotonicity so you put the two together that when we have stopped Tm is contained in this star So this is the smaller Tm remember Tm was the same as Tm plus 1 is the smallest solution of the recursive equations So we have achieved both things here We have proved that the smallest solution exists so the declarative semantics is well defined and it is obtained through this bottom-up evaluation So this is a very special case of the Tarski theorem It is the same argument that you used to give meaning to semantics to least fixed point logic but here it comes as a very clean as conjunctive queries plus recursive. What you get in the wash out of this that for every fixed program this is data complexity The bottom-up evaluation can be carried out in polynomial time Why is that? The reason is that the number of iterations is bounded by polynomial in the size of the database The degree of the polynomial is just the parity of the recursive predicate if we have one Every step of the iteration can be carried out in polynomial time because we do a relational algebra evaluation on some fixed query from a data log program So polynomial times polynomial you get polynomial By the way since we saw before that we can do data complex in log space Why can't we do this in log space also? What's the problem? But what we do in every step is first order operation on some fixed query Let me just say very quickly The reason is we have to carry the relational building along We have to store it As we will see you cannot do data log in log space unless very strange things happen So the bottom the bottom line is that the data complexity of data log is in P It's a very important result because you wanted to add recursion but you didn't want to get the complexity outside polynomial time So the data complexity of data log is in P The combined complexity however is high it's exponential time complete I won't show you this So there is a price that you pay there So this is the price of recursion if you will at the level of combined complexity Remember for calculus for first order logic it's P space concrete before the data log jumps to exponential time complete But data complexity is still in P time Let me show you two interesting programs two interesting data log programs because all we have seen so far is the transitive closure and this even and all Non-two-colorability can be expressed by data log program Why is that? Because non-two-colorability is the same as there is no cycle of length, right? So what we can do is take the previous program we had for computing on and even that's one of the reasons I wanted it So odd x, y and even x, y says there is a path for the length there is a path of even length between x and y and then we have another predicate Q with no variation a zero-ary predicate that is true if and only if there is an x such that there is a cycle of length from x to x So you can do non-two-colorability in data log Okay, that's in some sense a very simple data log program with a somewhat of non-trivial proof of correctness because you need to know the theorem from graph theory, right? By the way can you do as a sanity check can we do three-colorability can you write the data log program for three-colorability what would happen was that yes we would collapse NP2P but actually we know better from the work of ANUS that we cannot even do it we cannot even do it in least fixed point logic in fact what is IFP plus counting, right? So two-colorabilities and non-two-colorabilities is very special in fact that's where we draw the line Here is my absolutely favorite data log program is the path systems query Have you seen this before? Somebody might have announced this before So back in 1974 Steve Cook wrote a four-page paper called on a space-time trade-off This paper showed that there exist problems which are complete for polynomial time by a log space reduction So Cook not only gave us NP completeness he also gave us p-time completeness and one that he used to show that is complete for p-time is exactly the problem computed by this data log program Cook did not know that the log was not around at the time but Cook was always thinking in terms of automated theorem proving So to get a feeling about what this program is trying to do think of a proof system that has two parts A and has an eternally rule of inference So think r, x, y, z meaning that x is inferred from y and z using the rule r For instance, something like modus ponens Something like modus ponens if I have phi and phi implies psi I can get psi, I can think of this as being an eternally rule of inference that says I get psi from phi and phi implies psi but the solution is the same character So this program, if you have this interpretation gives you the theorems of the system tells you that x is a theorem if it is an axiom or you can get it from two other theorems using this rule of inference So Cook proved that computing evaluating this data log program is a p-complete problem Now what do I mean as a decision problem So I give you A and r and some value B and I ask you is B in T This is a p-complete problem So the data log can express p-complete problems I mean that's the bottom line The data log can express p-complete problems Okay In particular this shows you that the data log evaluation is not going to be in log space because it can do p-complete problems So in some sense even the data complex is higher still in p-time but higher than the log space that we had before which goes back to the remark that we had earlier Very quickly What is linear data log Linear data log is what you get is the fragment of data log in which in every rule you have at most one recursive predicate At most one So you can write cousin from sibling You have a parent You are given a parent You define sibling as having one parent in common and then you define cousin this way And so there are some very amusing things So when we had the election in the US in 2008 They found out that Barack Obama is an eighth cousin of Dick Cheney You can't think of two more different people than that When the link is life, I checked it last night It's correct, it's still there And if you think that this is not good enough Here is another one Sarah Palin and Prince Diana are tenth cousins So this is really also you shouldn't put bounds on it You should just let data log run until you discover all these things Anyway the story of linear data log is that it's very interesting The data log program is linearizable if it's equivalent to a linear program For instance the transitive closure divide and conquer is linearizable because equivalent to this On the other hand the transitive the program for the path systems query-cooks program if you will is not linearizable is not linearizable This requires some proof Of course from complexity reasons can be because it's easy to see that every linearizable program is in NC So you would hear that but actually you can prove it By the way, telling if a problem if a data log program is linearizable is undecidable subsequent versions of SQL 99 subsequent versions support linear data log A linear data log program is a program in which on every rule the right hand side has only one recursive predicate So here parent is given to us the recursive predicates are sibling and cousin the second rule has only one recursive predicate, sibling the third rule has only one recursive predicate which is cousin So that's a linear program This is not linear because the right hand side has two occurrences of the recursive predicate On the other hand is equivalent to a linear one But this the second one is not linear but is not linearizable Is that clear now? So that's what SQL supports Supports only linear data log programs I think so It relates more to the fact that when it's linear you can implement on a stack You can do recursion They were driven I don't think that the standard people knew anything about the theory about time-complete problems and anything like that They were just driven by how easy it is to implement linear So you can write this is the syntax I'm running late, I don't want to explain it more But that's how it is Let me I'm almost running out of time Let's take a look and try to tie this up with some other Let's compare the data log with a calculus So Unors of consigned queries are contained in the data log but calculus is not contained in the data log The data log cannot do all of calculus For instance, the reason is simply because we have only an out-monotone operator we cannot express even the quotient operation For instance, we cannot express difference we cannot express the quotient On the other hand the data log is not contained in calculus because we have a transitive closure So here is the data log and here is first order logic or calculus Certainly what we have here are the unions of conjunctive queries which means what else is there and the answer is nothing else and we owe this to Ben Rosman as a corollary to Rosman's theorem the preservation under homomorphism is that data log intersects with relational calculus is precisely the unions of conjunctive queries Why is that? Because if you have a data log query which is also expressible in relational calculus then in effect you have a first order query which is preserved under homomorphism or is equivalent to a union of conjunctive queries So in terms of expressive power this intersection this intersection is precisely precisely the unions of conjunctive queries That's exactly what this intersection is It's a nice way that's something that we really didn't know until actually we knew it another way which I'll show you now is a very simple corollary to Ben's wonderful theorem Here is another theorem that we can get very easily from what we saw here together with Ben's theorem Back in 1987 Aytai and Gurevich had a paper in Fox where they proved this theorem with a highly nontrivial proof and essentially they used bounded tree width and all sorts of other things and they said the following statements are equivalent for a datalog program Py Py is bounded Bounded means that there is a fixed number K such that on every database you only need to iterate this program K times You know what I'm saying is that your recursion is not needed It tells you that this happens even though the query is defined the query defined by Py is expressible in first order logic γιατί είναι ένα πρόεδρο για το σύνταξο εικόνα. Δεν πιστεύει ότι αν η σύνταξη είναι εξηγημένη πρώτα ή η λογική, δε θα υπάρχει ένα άλλο δεύτερο προγραμμό που δημιουργεί. Αυτό είναι το προγραμμό εικόνα. Τώρα, ένα από τα δύο σύνταξη είναι δυναμή, αν η πιστεύει, όμως είναι ένα γραμμό της σύνταξης, ελπίζω να πρέπει να κρατήσει. Οπότε, η ενδιαφερή στις σύνταξης είναι δυναμή. Αυτή η διεθνή που μεταφέρει με ένα στιγμό, θα χρησιμοποιήσουμε το Ροσμάν's θεόριο. Αυτό το διεθνήκε για ένα. Αν η Q είναι πρώτα ή η διεθνή, αφού η στιγμή είναι προσέβαιη από τα στιγμή, είναι ένα κοίυελαντο στο στιγμό της κοίρες της κορυνής. Βλέπτες τι έχουμε τώρα. Στο δίκιο έχουμε το στιγμό της κορυνής κορυνής, διεθνήκε από το Ροσμάν's θεόριο. Στο δίκιο έχουμε το αφού της κορυνής της κορυνής, που θα φύγουμε από το δεθνό της κορυνής. Τώρα μπορούμε να χρησιμοποιήσουμε το Σαγγύβια Νακάκη's θεόριο, που σας είπα πριν. Αν έχετε δύο κορυνές της κορυνής της κορυνής, then every member of one contains the other, then every member of the left contains the other. Αν αφού υπάρχει το Σαγγύβια Νακάκη,εξάρωση, ότι αυτή τη κάνατο β 싶 απορραμbucks, πρέπει να καιγησούει ένας κορυνής κορυνής της θεότητας, πέγω raised endurance fills από κάνασ yaptι τέτοιο διαliesο亲άτας. Κ완 και το Σαγγύβια Νακάκη, αφήρτο μετά το β πρόγρο της αλλά αυτά, είναι ανέγρασibocarpibrinsky Ωγσίο για να ξέρει λύissonδο της. Πρόιχέται εντήpielικο και βiphopar crews, εκ τα ανα floss Singingmin's σταν ταξа, εντύπεσπου από τη γιαταγή του στημοισ convictions. Υπάρχουν αξιόδιες της Δέταλογκας με εμφανιστήτητα που μπορεί να μην να μιλάω για αυτό. Δέταλογκη με εμφανιστήτητα στην Ευρωπαϊκή Βαθμότητα. Βέβαια, αυτή η αρήση ήταν δύο κερδίδες. Σήμερα, στην τεστορία του 1980 και την 90' δεν υπάρχουν πέρα. Έχεις να μην να μην καταλάβεις τα σημαντικά, τα στρατηφότητα, τα σημαντικά, τα καλύτερα, τα σημαντικά, τα σημαντικά, τα σημαντικά, τα σημαντικά. Αυτό δεν μπορεί να μην καταλάβεις. Η 15η του Βιαμπιτεύου Βιάνου Βούκ έχει μια καλή επαγγελή. Θέλω να τελείωθα με τον σχέδιο. Έτσι, έχουμε αξιόδει Δέταλογκου, να καταλαβαίνει την εξακρίναση εξακρίναση. Έχουμε δει σε κάποιο συγχώρηση, το κοίτευμα της ευαγελίας, το κοίτευμα της Δέταλογκας. Ποτέ για τον κοίτευμα της εξακρίνασης. Βέβαια, υπάρχει πόρυ αγγελία εδώ. Πόρυ αγγελία είναι ότι ο Δέτος Μουέλι, το 1987, στην Πόρυ αγγελία, που, όπως είχα πωθήκα, είναι το πόρυ αγγελία μου, διότι η εξακρίναση εξακρίνασης για δέταλογκου είναι αγγελία. Εννοείς, είναι αγγελία, επειδή είναι αγγελία με ένα μόνο εξακρίνασης. Είχε διάφορες εξακρίνασης και εξακρίνασης. Η εξακρίναση είναι πολύ συνδυνή, επειδή μόνο εξακρίνασης, επειδή είναι δύσκολο με ένα γένος της εξακρίνασης. Αυτή η εξακρίναση σφιχωριότητας, είναι ένα εξακρίναση μονοτήρας. Είναι ένα δύο εξακρίναση σφιχωριότητας, με τη δύο που εξακρίνεται στη ίδιση της two-condex free grammars, και δει με μια εξακρίναση αυτή την προβλήματα για το δέταλογκου. So the picture I want to leave you with, and this is essentially the last slide, is we've gone started with calculus, we dropped to conjunctive queries, we can go to unions of conjunctive queries and not get anything worse. The power of recursion is, this is the price we pay for using conjunctive queries. Η στιγμή στις栏δην θα σέγνει στην εξεντρικία ψαρφή, η στιγμή ρεύμα της δεσμότητας είναι η πραγματική ασκότητα που behind and condem, να κάνει μέσα. Λοιπόν, ελπίζω ότι που έχουμε κάνει στις εδώ είναι να δείξουμε ένα σ fuq μέσα η Ευρυπία Βινόυα ήταν πολύ νευαδικο, στις λέει ότι η θέα τέτοιμη ρεύμα και την προοδική χαρνή θεία είναι πολύ σημανή. Η θέα θέα τέτοιμη ρευμα είναι η δεύτερη αφή, η δεύτερη εργασία, αλλά also the database provides the concrete scenario of finite model theory. I mentioned yesterday that, to me also, it's a case of logic from computer science. I mean, the catalog is a case of logic from computer science. So thank you very much for your attention.