Hi everyone. Welcome to the Jenkins online meetup. Today we have a presentation about moving from local installations to Scalable Jenkins and Kubernetes. We will talk about the Kubernetes Jenkins operator. Thanks to Mateusz and Peter for joining us. They represent Kirtoszlaw, one of the main contributors to the Jenkins operator on Kubernetes. And yeah, today I hope you will have an interesting presentation. And if you haven't used the Jenkins operator before, if you haven't used the Kubernetes operator before, I believe there will be a lot of good topics to study. Quickly about the Jenkins online meetups. So Jenkins online meetups, anything about Jenkins, including Jenkins for users, Jenkins for developers, Jenkins for the community. Basically, these meetups are quite informal. So we for discussing topics we have tried to have live Qoomi live demos. We focus on the interaction with users. We will have Qoomi and we will have Afterparty. Well, a kind of Afterparty after discussion. And yeah, you're welcome to participate during the presentation, after the presentation we will again share chats where you can join and ask the questions. As you may have heard, we do a series of Jenkins and Kubernetes meetups. We started the session last year, we had something like six meetups during 2020. And we continue this year with this is our second meetup. This is a meetup about Qoops and Jenkins. And we will be talking about Kubernetes a lot during CDCon and during the contributor summit on Jenkins 25th, sorry on June 25. So feel free to join. And also there is DevOps world, which is done in September 28th 30s. And you're welcome to join. So big line for call for papers is actually today. And the program community there. And if you're interested please play a few weeks ago we did presentations about how to apply and what will be the community agenda DevOps world. So just consider that. And yeah, shout out to the sponsors of the meetup. So it's continuously different foundation which sponsors the platform and called this which sponsors time for horse and also shwack, etc. So, let's move on. And yeah, for Jenkins and Kubernetes we always look for speakers. So any experiences about Jenkins and Kubernetes relations are welcome so how you use Jenkins and Kubernetes how you do automation how you develop your pipelines. And if you develop plugins or if you develop things like hand charts to Jenkins operator you're more than welcome to join and present. If you just, just doing integration is Kubernetes tools you're also welcome to join so basically be interested in the entire cloud native system. So you may have seen this huge diagram on the CNCF. So basically whatever they're not just Kubernetes is a subject of interest for us, and you're welcome to present. So there is a link below I will share later in the chat. So you can just submit a talk and we will be happy to host this meetup. And of course, today we have two hosts. So it's me. My name is Alek Timnash. I'm one of Jenkins contributors. I work on many topics and I also have a nice online meetups. And we also have Rahram who joined us today. Please welcome him. Would you like to do a quick introduction as a host. No. Okay. Okay, so let's keep working through the data and yeah. For Kone, for Kone, we agreed that the questions will be answered after the main part of the presentation. If you want to ask questions please use zoom Kone or the chat. If you want to ask to use zoom Kone, you can see the controller in the bottom there is Kone section. Ask any question and we will either ask unsighted synchronously in the chat or ask speakers after the presentation. So the main part of the presentation is stay online because we will have an open discussion. So we will stop the recording. So any questions you would like to ask related on related you can ask them of the record and we invite speakers to stay as well. Yeah, it will be very informal and you are welcome to participate in this part as well. After the meetup, there is Jenkins operator on the virtual slot, I will share the link in the chat. So you can join this channel and ask any questions about the operator there and the entire contributor community is in this chat so there will be many people who are interested. So during the meetup, please keep in mind that we have a code of conduct in Jenkins. This code of conduct is very simple. It's been nice. And yeah, there is more text here but it's quite short. So I won't stop much on that. Please welcome Mateusz and Peter. They will do the main part of the presentation. Thank you. Thank you as well for the introduction. Thank you very much. And let me just share my screen. Okay. Does it work? Can you see it? Yep. Yeah, thanks. Great. Thank you. Okay, so welcome everybody to our presentation about a little bit of a journey from a local Jenkins installation to a Kubernetes cluster deployment. In this presentation, we would like to show you how different size organizations might deploy Jenkins and the challenges they might be facing on the way. But first, let's talk a little bit about ourselves. Okay, so my name is Piotr Reba. You can call me Peter. I'm a DevOps engineer at the virtual slot and together with me is Mateusz Korus. Matt, would you like to say a few words about yourself? Yes, hello everyone. My name is Mateusz Korus and I'm very pleased to be here and share the presentation with all of you and as well as Piotr as the co-presenter. So we have some interesting stuff prepared for you and we hope you will like it. Okay, thank you very much. Yeah, so this presentation is split into parts. We'll be talking about the challenges with different ways of deploying Jenkins. And Matt will talk about how Jenkins operator can help you when using Jenkins on Kubernetes and about some plans for the future. So let's start our journey with a sample project. Okay, so say hello to Alice. She has a business idea. It's a mobile app that helps you keep a healthy diet, but to be honest for our purposes, any project works. And so she's written some code, has some tests, and she would like to integrate a CI into her repository to build tests and upload the staging build. So naturally she looks at Jenkins. This is an option that will get the job done and what is more, it's free. So yeah, great. That's the app. Okay, so Alice would like to keep things simple. So she decides to use Docker. She runs this command and she opened her browser. It worked. This is the initial configuration. And yeah, basically after setting an admin password. She saw this and empty Jenkins instance. And so she did some more configuration on her jobs and pipelines. For example, she might have used the Jenkins file. And then she finished her day and went to some pierogies. So basically she went home. So the next day she comes to work. She puts up her computer, realizes that the Jenkins is down. And so she runs the same command again and opens Jenkins and more or less sees this. So again, empty Jenkins instance. It's gone. So, well, it turns out you can't really lie on the same container to be always up. And you have to treat containers as short lived and make sure your data survives through container recreations. So Alice does a little bit of more reading on the subject and decides to change her Docker run command a little bit. So there are two changes here. First of all, the minus the option, which will start Jenkins in detached mode. So it won't block terminal window. And the second change, she will mount Jenkins home. So this way, her data will be safely kept intact on her local file system. And to be honest, you might call it good enough at this point. You might want to tweak it a little, especially if you're a fan of declarative configuration, you might transform that into a Docker compose file and apply that. But we're not done yet. So Alice realizes she has a lot to do in order to get her product market and she doesn't want to miss her opportunity. So she asks her friend Bob and to help her with the app. And at this point, it starts to become a little bit problematic for them to share the same CI. She might be working from different places, either, well, the same city or even around the world. And also how this notices that when jobs are run, her machine slows down so they figure out that it will be best to move their CI CD tool away from developer machines. So local installation suddenly it's not very practical at this point. So would you move it? Well, one option would be a thing called the machine. So basically, you take what you've been running on your own computer and move it somewhere, somewhere else, wherever it's handy. And by doing that, you can allow more people to use that software since it's not tied to your computer being on and around. So this is an old PC and mini computer like Raspberry Pi, just bear in mind that Raspberry uses different architecture so you have to use different docker images as well. But whatever that has enough power works at this point. And I'm sure many of you have seen this kind of deployment in their careers. It's a little bit of an improvement than we had before. It works especially if one team basically works from the same place, same room. And what's more, you don't have to think about exposing your instance anywhere outside or figure out any security issues. But this approach still has some flaws and well to quickly grow go through them and not to go into too much details. So first of all, you have to do system maintenance, keep your OS in good shape. You have to take care of hardware. When it fails, it will fail eventually. So prepare for that. And do some upgrades in case you run out of processing power. Some environment environment issues might happen that could range from broken AC and your machine overheating or to somebody unplugging your machine from the electrical socket. Yeah, and you have to deal with it networking and security risks. You have to figure out how or do you even have to expose it outside use a DMZ VPN or whatever you have to figure that out. And lastly, it depends on the SLA of the electrical power and the internet supplier. So in case your internet is down. You might not be able to access your CI. So the question would be to use a dedicated server. So question is, should at least now spend a lot of money and order some servers and start looking for a collocation options. Well, it turns out that probably not. Unless she wants to pivot her business. Or that she just wants to build her up. So she'll probably pick something from any cloud provider. And this way she can get rid of a lot of environment problems. And she doesn't have to worry about hardware anymore. It will be kept power to connect to the internet by the service provider. The hardware will also live in a restricted area so somebody stealing your hard drive is much less likely on display. But if we were to revisit our drawbacks from before. So system maintenance stays we still have to do this. Hardware failures will be handled by the service provider. You still need to upgrade at some point in case your Jenkins has to run more jobs. Environment issues get cut down by a lot. Still networking security topics. And lastly, you will suddenly not really depend on the SLI of the electrical power and internet supplier, rather the service provider so the server itself. And yeah. So, if we are to identify a single point of failure at this point, it will most likely most likely be the server itself so in case it's down the whole CICD services down. And so the next step would be to try to address this. We'd like to run our Jenkins in some sort of a cluster and well, normally, this would add another complexity of managing more servers than before because well, we had only one and now we are thinking about several servers so it would be really nice if we got the benefits of running a cluster while building being abstracted away from the OS level. And if you even could get this as a managed service would be a huge plus where all you do is say I want to run this application or to be more precise I want to run this container image. So let's imagine a world where if a container fails gets restarted automatically if an underlying server dies, for example, its work gets rescheduled on other servers automatically. And a world where you can scale your application automatically based on your resources. So, as you probably know, we live in this world. Yeah, that's Kubernetes. That's the description of it. And so I think we can all agree that Kubernetes gives a lot to the cloud computing world. It changes the perspective. That's a little bit of complexity, but you can't really avoid that, that because it abstracts away the underlying OS and hardware and software. And what's more, it's available as a managed service from all the major cloud, public cloud providers. And now we are not really thinking about servers but see Kubernetes cluster as a whole. We are no longer doing with OS and services running processes. We're just thinking about Kubernetes resources like namespaces, spots, services, persistent volumes. So, yeah. And that's how a simple manifest file of a Jenkins deployment on Kubernetes might look like. There are some apparent problems with that. First of all, Jenkins will run in the default namespace and really you should, well, it's advised to have a separate namespace for Jenkins to do some separation of resources. Secondly, and probably the most important aspect, you won't be able to connect to the Jenkins instance because it's not exposed outside of the Kubernetes cluster. So you would have to work a little bit more on that, either your support for wording, set up a service, configure ingress, these kind of things. And lastly, it has the same flow as our initial Docker run command, which is the Jenkins home is not mounted anywhere. So your data will be lost. Well, most likely will be lost. You can get to it, but it's not easy when people gets restarted. Yeah. So, so far, we've been mainly focusing on availability issues. And there is more. There is whole maintenance and operations aspect to that. So we have to configure your instance, you have to find some plugins installed and keep them in good shape. You probably want to run some scripts. And there is a whole topic of observability and monitoring in basically if something bad happens, you have to know where to look for issues and know how to address them. Maybe set up a monitoring and other things. So you'll be the first to know that your services down. And the security hardening. And I would like you to refer to the Jenkins documentation on that subject because it's quite extensive and that's, well, that's that's an important part of documentation to it in my opinion. Yeah. So, I think we can all agree that using Kubernetes will put us in very great good place to begin with. But there is more, more to deal with. And we have to find a way to glue things together so Kubernetes configuration and maintenance of Jenkins and security aspects, all of that. I might ask a question. What is one of the sources of our problems when it comes to running Jenkins and Kubernetes. Well, it has state. It's not a stateless application. So it's not like a pure mathematical function where the output depends only on the input. But Jenkins has its states. So it in the west. That state influences the way Jenkins behaves. And I'm sure many people have experienced at least some problems with Jenkins that are caused by this. For example, one plugin update my lead to a thing we call dependency hell where you have to figure out how to update your plugins and fix their configuration in order to for Jenkins to work. Yeah. So how can we solve this issue. So when it comes to configuration, we can use groovy scripts, which is basically the main entry point to the Jenkins API besides rest API. And this allows us to execute groovy scripts directly on Jenkins. And there are a few ways to do this. One of them is to put some scripts in Jenkins home in the special directory, and they will get executed on each Jenkins start. And I would like to refer you to there is a github repository open source when where there are some very useful groovy scripts that are just waiting for you to go and grab them. So, yeah, feel free. And as a side note. Yes, you can extend Jenkins. And that's a really powerful way. You can use one of the 1500 plugins that are available to download on the site presented on the slide. And just you have to be careful to quote with great power comes great responsibility. So you wouldn't want to end up in the plugin dependency hell. You have to find a balance with the plugins you want to install. So let's mention some plugins that were mentioned for our case. So first of all, Kubernetes plugin. And that's a great way to integrate Jenkins with Kubernetes in order to auto scale your Jenkins agents on demand. So you basically define a pod template and keep your Jenkins main instance running. And when the job, one of the jobs needs to run with the triggered somehow. And a new pod will be started using the pod template. And after the job is done. It will be cleaned and the resources will be. Well, received back pretty much. Yeah. And another topic that will help us with configuring our Jenkins. Well, basically those two plugins configuration as code and the job DSL. So configuration as code. Let us configure Jenkins. And it's plugins using YAML files so human readable files. And the job DSL uses a specific domain specific language that lets you define jobs using again a file. And so it would look a little bit like this. So you will probably have a git repository where you would keep all your configuration files. And by doing this, you get well bashing branches, you know, all those things for free to be honest. And yeah, so we would like to have a recipe to run Jenkins using job DSL and configuration as code kept somewhere. And point Jenkins in that direction. So it uses that to configure itself. And there's another subject that we haven't mentioned so far. So it's the authentication and authorization. Well, as a starting point Jenkins comes with a username and password. Authentication, which is good. It works. But if your organization starts to grow, you might start looking at different options. So for example, you can use a directory service like LDAP or Active Directory or use an external identity and the identity provider to set up a single sign on. You can integrate with external service using all of OpenID Connect. And when it comes to authorization, the builds in way is, well, enough to start with, but if you find that you need time grain access control, you might want to look at the Matrix authorization strategy plugin, which which is really cool can be configured using a configuration as code file. Okay, so a little bit of a summary. So we came a long way from local to Kubernetes deployment. You might be hopping on the same journey at different places. You can start with Kubernetes, which will give you a lot. But I think we can all agree there are still many things to start out to have Jenkins runs mostly on Kubernetes. And it may be important to mention that we haven't touched the subject of backups so basically anything that is not configured using files will have to be backed up somewhere and reverted. And the goal, the goal is to be able to start Jenkins in a predictable and consistent way. And that really concludes my part of the presentation. So Matthew, it's your turn, the stage is yours. Thank you. Great presentation on some of the real problems that people trying to deploy Jenkins and Kubernetes phase. And here comes the Jenkins operator project. The Jenkins operator project is all about automating the lifecycle of Jenkins, which itself is an automation server so you can see that it gets pretty meta. But since we all here love automation, I believe that everyone here is excited about the possibility to have a reputable and predictable deployment of Jenkins possible to them on any cloud infrastructure really that supports Kubernetes which is like almost all of them. Next slide please. So a sneak peek at the Kubernetes operator architecture, the Jenkins operator architecture. You can see here that within the Kubernetes cluster there are two parts to namespaces. The first one on the left is the namespace of the operator itself that works by managing the lifecycle of the Jenkins, which is surrounding the separate namespace on the right. The Jenkins in the right in the right on the right part is just a complete deployment. It has the Jenkins master pod Jenkins agents as well as the jobs. And it's fully functional by itself, but all of the upgrades scaling or scaling is actually done by the Kubernetes plugin that operator uses and provides to you by default. And besides that there are two parts which are the external storage where you can backup your artifacts and your job history, and also the ingress to Jenkins, where you might want to set up VPN or a bastion host. And with Jenkins operator it's very easy to set up identity provider, such as GitHub of. All of that uses all of that within the Kubernetes cluster also takes advantage of the airbag model, which is the model that admits privileges to certain users and group of users. It is very well built it's very rigid and it's great to use. Next slide please. So with Jenkins operator we are really trying to push for the configuration as code mindset, because we do believe that it's great to have your configuration as code. It allows your infrastructure it allows your configuration to be versioned and to be repeatable. It allows you to use the GitOps model which is the new hard work in the, in the industry. And it is great for debugging for understanding the whole infrastructure it's not just, you know, someone some somewhere, someday clicked a few buttons in the UI, and now nobody know how things work. So this is clear in the code that you can see in your git repository, as well as the history of how that state came to be. And next slide please. So as Peter mentioned earlier, one of the greatest things about Jenkins is its accessibility by plugins and Jenkins operator of course allows you to use plugins that you want to use. So if you have an example of a manifest, you can see that the kind is now not not a pod with just the Jenkins image but it's kind Jenkins, which is custom resource definition provided by the operator. And you can specify as many plugins as you want. So you have to specify the plugins that the Kubernetes operator uses by by itself such as the DSL and Kubernetes plugin. They come bundled with the operator cover can extend it however much alike. And the command to apply it is just as easy as it is for the deployment using a pot without using a Kubernetes operator. Next slide please. So as Peter mentioned earlier, one of the big subject for Jenkins is security Jenkins is by software and software standards, a pretty technology with pretty long history. So it has some legacy mechanisms with it built in it that even Jenkins themselves, the Jenkins itself advises users to disable such as old JNLP protocols, CLIs and other ones. So Jenkins Kubernetes with its initial configuration of Jenkins does it by default so we don't have to even worry about it. And you will, you will have a secure Jenkins deployment without having to manually tweak every setting that needs to be tweaked. So the top of that, as I mentioned before, it leverages the airbag model from Kubernetes, which is really good and the, like the new industry standard really. So that's, that's a great thing about the operator it, it allows you to have peace of mind without actually doing very much work in the department of security. And next slide please. So the thing that Piotr mentioned before also is backup and restore with our mindset with our configuration as code mindset. The only thing that you need to backup are jobs history and artifacts, because the whole configuration you have in your Git repository. And whenever Jenkins starts up, it can just pull all this configuration and have all your jobs ready. So you can have a restore script that runs and pulls your, pulls your jobs history and artifacts, but the job definitions themselves will be handled by, by the operator using the job diesel plugin. And next slide please. So, if you stick to, to some of the good practices that we, that we presented, such as having, you know, an fMRL state of Jenkins, the repeatable deployment, basically having a recipe for the Jenkins that you want to get. You can use the Jenkins operator to work behind the curtain to actually to actually deploy the Jenkins in the state that you specify. And it handles all the updates, all the lifecycle that needs to be handled because running Jenkins in Kubernetes is not a trivial task but with the help of the Jenkins operator we hope to help it become at least less of a hustle and more of a pleasant thing to do. Next slide please. So what's on our roadmap. And that actually relates to a question that was asked by Thomas Love about the new release of the Jenkins operator. We are working on a new schema for the Jenkins operator. And it will introduce more granular, granular elements for the components of Jenkins. So, we are migrating from pod to deployment as, as Thomas Love, as per Thomas Love's question. We want to refactor the end to end tests and we also want to be more engaged with the community so we need to establish some processes within the Jenkins operator to help you basically work with us. And also on our horizon is, is there integration with third party systems. And next slide please. So, as I mentioned before, the biggest problem with current version of Jenkins operator is that it only uses one custom resource Jenkins custom resource. And we are actively working on a new API that will allow for it to be multiple custom resources such as Jenkins Jenkins agents. And seed jobs. And because of that, it will, it will unlock many possibilities for future features and, and upgrades to the operator itself. And so, because not every element is dependent on every other element. The updates will be less of a hassle, not everything will have to restart at the same time, but it is also very difficult subject to tackle. Maybe on the right we have been working very hard to actually figure out what the dependencies between between elements are and how, how we need to go about, you know, there are interactions to make it all work seamlessly and have as much as much as possible. And also we are actively getting involved with the community as of this presentation and also actively looking for feedback you can see the link on this slide. If you are using the Jenkins operator or if you want to try it you can send us our feedback, it will be very much appreciated and it will help us shape the future of the Jenkins operator, hopefully in the right way, thanks to your suggestions. Next slide please. A big part of how we are proud to be working in the community is that we have become an official sub project of Jenkins. You can see the also the link on the slide with the announcement on the Jenkins page and we hope to be engaged with the community and hopefully have a very good and beneficial relationship in the foreseeable future. Another thing that we're doing for the community is that we are participating in the Google summer of code which is, if you haven't heard of it, a program from Google that takes some issues from open source projects such as Jenkins or Jenkins operator and issues or, you know, features that need to be added. And it gives some students opportunity to work on those features. And it's, you know, you probably remember how difficult it is to get your first experiences as software engineer so we think that this initiative is great for for students to have their first, you know, real life experience and also help the open source as a part of this project we are implementing a security validator. That will help to for the end user to see security problems with the plugins that they are installed because, you know, if you have, if you have some so many plugins there are so many things that can have some vulnerabilities and it's important for the for the user to be at least aware of them. And you can read more about this project on the link at the bottom of the slide. And next slide please. That's the last slide so thank you very much for being here, everyone. Thanks for co presenting with me thanks to all I can background for hosting and questions that we haven't answered it. Yeah, just getting started. There is a lot of questions. There's one thing I would like to mention. And there will be some t shirts as prizes for the feedback form. And one of them, one of these, basically, I'm not sure if you can see them. Jenkins operator t shirts. So, yeah, yeah, we'll report the link because yeah I'm spamming everyone is links at the moment sorry about that. But yeah, repeating the link to teachers is important enough, that's for sure. The first matter that I tell them but I don't feel like I've answered fully is the Thomas loves question about the new release of the operator. The new daily release that new like mainline release we don't have an official term for it yet will be when we're finished with the new schema when we have tested it and it's good for for you to try out as well. But we have also introduced a nightly builds, which are built daily from the actual state of the repository so any current fixes that we are implementing in the old API are available through the nightly builds and you can check out documentation on how to take advantage of them. Okay, so next question is about the using a single body set of deployment that's what we're working on. And hopefully soon it will be a deployment. Okay, Kubernetes Jenkins operator replace a Jenkins server external. I'm not sure about it. What's the question. I mean, a question from corporate training. I hope I pronounce it more or less right list. Oh, sorry. It was me who kicked that. So, why is Jenkins run a single port instead of a deployment in the operator. I just said that currently, well I don't know why it's because it's, it's been done this way while we were implementing the first version of the operator but we acknowledge there are some some issues some inconveniences that come with it and we're working on having Jenkins as a deployment and not just as a boat so I hope that answers the question well it doesn't answer the why but I hope it's less relevant. If you want to proceed to the results. I did have issues for the operator where you can raise any questions feedback and the race feature request if you have them. So I can share the link later but yeah this is the best way to actually post any request concerns because that's how open source works. Of course. And yeah another note, I shared the link to the feedback form in the chat. So we would appreciate your feedback so that we could make our meetups better. So that is feedback to organizers and also feedback to speakers. And we will appreciate all kinds of feedback and process that and maybe share it with everyone. So please send your feedback from because we really appreciate the feedback and we really process the feedback. Yes feedback is also very much appreciated all the time. It helps to shape the the future and see what can be improved. So maybe Piotr you want to answer some questions from the Q&A. Which one. Well, your choice. Have you tried to run this using github approach from Thomas I don't think we answered this one. But yeah, we basically are trying to enforce github approach with using the operator so that's the advice way to do this. So, which questions haven't been answered yet because it looks like I might be behind on questions. So I have a question about Covernet Jinx operator replace a Jinx external service I believe be answered, right? Yes, I think yeah. Okay. Can configurations code be used on an already running environment. You didn't have the plugin before to migrate over configuration. That's an interesting question. Of course you can install a new plugin. Well basically configuration as code. You can add it to your Jenkins. But I'm not sure if there is a way to export your current configuration as a YAML file that will be compatible with configuration as code. And also, well, what I would advise you to do to maybe try to spin up a test Jenkins instance. You can start with this plugin and gradually try to well in a safe environment and gradually try to migrate your configuration. And well, basically, if something goes wrong, you can always start from the scratch because you won't be touching your main Jenkins instance. And one thing is that you want to keep a recipe on how to get to the desired state of Jenkins. And unfortunately if you have a Jenkins that you just clicked some options in UI it might be a challenge to to migrate it to configuration as code or if you do that, and you spend some some time on it. We do believe that in future it will be worth and it will save you many headaches in the long run. So the next question is about OpenShift. So the question is from Andres, will the creator be available on OpenShift? Well, it will. The idea is, I was with, but with the changes we are currently doing the big API schema change that will be probably released without OpenShift support, but I'm not sure at this point. The OpenShift support will be added eventually. So currently, as it stands, it should be working on OpenShift. But we are mainly focusing on the new API and trying to get on Kubernetes and OpenShift will come later. One thing to mention that actually in Jenkins we have two official operators, because at some point the Red Heart team started implementing their own operator. And basically we ended up this creating a hard fork for that. So currently if you go to the Jenkins CA you can also find the Jenkins automation operator, which is basically being maintained by Red Heart and they have implementation specifically for OpenShift. So as a user you have a choice which project to take, you can take the Jenkins operator, you can take this Jenkins automation operator. And hopefully at some point today we will still converge into a single project, but it depends on many factors. But I just wanted to say that such option is also available. Okay, great. Okay, any other comments or more? Okay, so the next question is Jenkins operator production ready or like for 500 users? Well, I would answer this, maybe like this. It's as production ready as Jenkins itself. I guess that will be a big change. So it's always a good idea to test this first and maybe gradually move your users and basically test it out to see if it really fits your requirements. So, yeah, but, well, I don't see why it wouldn't be. Just bear in mind there will be some API changes. So maybe hold on with the production migration, just a little bit more. And about the scale of usage, well, I don't know if you have actually 500 users working on a single project. And with Jenkins operator you can easily spin up multiple instances of Jenkins. So you can have like one Jenkins per team or however you like to split it. And I think it will solve, if you have some issues with its scalability, I think it will help you to solve it by running just multiple instances of Jenkins and also Jenkins operator allows you to have the agents being scaled automatically using the Kubernetes plugin. So I think it is feasible to have 500 users using Jenkins operator but maybe not like one Jenkins operator. I would advise you to maybe consider splitting your workflows into several smaller Jenkinses. So that has the advantage of, if something actually goes wrong, it only affects like one team that actually broke something instead of all of your 500 users. Yeah, that's actually a very interesting point. Using operator makes it easy to basically spin up multiple Jenkins instances. So it may make your life easier but by well and easier, easier to scale with users. So it's easier to probably manage one small or few small smaller Jenkins instances that one done one huge Jenkins. So it's very horizontal not vertical. That's our approach. Next one is one. Sorry. Actually I had a question about production readiness and the scalability as well. So what I know, VirtusLAB created a product managed service based on the Jenkins operator. Do you have some metrics or the insights from these products in terms of scalability, etc. So what's your experience there? Wow, that's an interesting question. I don't have this data, but we can dig a little bit, maybe ask the right people and we can get back to you later. If you want to quickly speak about this product, it's also perfect to find. Or should we just move this coin? It is then. So, yeah, there is long question about testing the operator. So my suggestion would be to pass on this question until the live Q&A part so later on the record so that you can discuss it. And yeah, the next question is, could you please send us the link to the sample groovy code repository. Okay. I will. I just have to look it up. So maybe you can go to the next question and in the meantime. Thank you so you can just put it to the chat and share with a lot of this. Okay, so that is another question. Can I replace my actual Jenkins server on my visual machine with Jenkins operator in Kubernetes today. Yes, with Jenkins operator you get a fully functional Jenkins alongside with some plugins that we are using for it to run smoothly within the Kubernetes. And there's no reason why you wouldn't. Thank you. And have you ever looked at integrations with Qwirt. So since we are talking about visual machines, maybe it's something that was considered. I don't know anything about it. Hmm. What is the question. Yes, so Qwirt is a project. Well, one of my CNCF projects in incubating. So basically what it does the users Kubernetes device plugin API, and it allows to provide a specific virtual machines as Kubernetes resources, so that you can acquire them. So I had an experiment when Qwirt was used to provision agent of the virtual machine and share between controller instances. So something like that. So I guess the answer is no you haven't looked in that. But yeah, general it's a cool feature it's a. Yeah, we're certainly going to write down and look into it in the future so thanks for the suggestion. So, yeah, I actually have seen one implementation so maybe there will be a presentation later about that. Let's see. So, I am moving on. So, yeah, there is a question I didn't quite get it. So let's turn it to you. Yeah, the next question is from jk if it's a single port and not a deployment the operator, make sure it's running. If I know to this. This part is running colon gold dance for example. So basically what is failover policy for the current operator implementation. Yeah, that's what we'll get we started to put it shortly. So that's right. Yeah, remind it to everyone. We will appreciate your feedback and the feedback form. So, yeah, I'll share the link again. We will appreciate your feedback. Okay, so we still have some time. So the meetup was announced for 60 minutes but we can continue a bit I think is it fine with everyone. Okay, so let's continue a bit and if you have to drop at this point. Thank you and we will publish a recording later today. So, yeah. So, the next question is about notes so can one at custom Jenkins notes. For example, notes which are not containers. I guess it's a bit aligned with what I said about cube dirt. But yeah, what's your opinion on that. Currently, I don't think you can. But it's something that we could think about at some point in the future. The problem is that the agents are contacting Jenkins currently through to Kubernetes services and the services may not be exposed outside. So, as it stands, it won't allow you to connect your agent from the outside. So, yeah, actually, in Jenkins, there are two types of agents that are involved agents and about engines. So in bond is those ones which initiate connection on their own, for example, classic general P agents, though they don't really use general P is standard. And if you are the types of agents like Kubernetes agent, but at the same time, they're also outbound agents. So for example, SSH built agents, my agents and other plugins, for example, CC to plug in allows to connect the engines in the in bound mode over open SSH. So for these agents, you can actually use Jenkins operator because whatever only thing you need is to put configuration for this agent in the configuration is called YAML. And once you put it there when your agent starts up, these agents will be connected and then Jenkins will manage them as outbound. And the agent in this case Jenkins will be connected to them. So even if they're not within containers, but it doesn't cover all use cases for agents and Jenkins. It's still quite, quite handy. Yeah, but the terms of operator there may be a manual workaround possible. Yeah, so you can try to do this and look it up. It may be possible to do this is just not supported by the operator by default. So there may be some manual work involved there. Maybe another example for the future request. So outbound engines. So there are some examples in the Jenkins file runner documentation. And in the Jenkins configuration as code plugin documentation. So basically all these cases can be reused in the Jenkins operator. So of course you wouldn't be running Jenkins file runner. But yeah, the same configuration can be used to and if the plugin is present, you will be able to connect an external agent. Okay, moving on then. And yeah, thanks. So another one. I believe you answered that. Yeah. So, yeah, the question from India does the operator include actions like requesting storage from your cloud provider, the volume from API. And if so, how is it restricted to certain quite product providers. So that's something that's supported natively by the we are basically just making persistent blank claims. So there's there's this handle by Kubernetes. We are just interacting with the Kubernetes API to make those claims and the underlying you know, actual claiming happens by Kubernetes itself so we don't. We just try to leverage as much as much power of Kubernetes as possible and you know not to reinvent the wheel. That's a totally right approach. Another approach which is technically possible in Jenkins and in the Jenkins operator there is a plugable storage set of stories. So I've seen some of them. There is a link on Jenkins. So for example, there is a plugable storage available for artifacts for test results. And again, it's just a configuration and the number of plugins installed so you can use them along with Jenkins operator. So for example, there is actually for managers for S3, which integrates with API's and even some data from agents without passing through the controller. So there are implementations like that and again, they can be used in Jenkins operator. But yeah, I wouldn't recommend that if you can use a Kubernetes native way. Yeah, it's probably important to mention that we want to be portable so we basically want to have a Kubernetes as a requirement and we want to run on different types of Kubernetes clusters. So we have to keep this in mind that, well, other usages in mind also, yeah. And speaking of external storage, from what I know Jenkins operator includes external credential storage by default through Kubernetes secrets. Is it correct. Yes, we are using Kubernetes secrets. And there is a plugin that basically ports Kubernetes secrets into the Jenkins credentials. So yes. Thank you. So it's quite important part and nice that it's supported natively. I'll post a link to this plugin later. So the next question. Is it possible to spin up several Jenkins instances using different configurations of Jenkins. Yeah, I mean, okay. Okay, so I can answer this question. One operator only can control and provision or one instant of Jenkins, however, you can spin up as many operators as you want so you can have multiple instances of Jenkins, but that means multiple instances of the same operator itself, which have an advantage that you that you can use, you know, different namespaces for it and have that level of isolation as well. So, yeah, I hope that answers your questions. Yeah, and it might be something to consider for the future to support more Jenkins instances for one operator. So, yeah, what do we have next. So the next question is about, you mentioned that the one Jenkins controller per team setup, do you have any suggestion on how to manage common resources, most available licenses in our case, and local resource plugin be used and think that cross multiple Jenkins controllers. I can actually provide the answer, but you may not like it. So, no, you cannot use a local resources plugin in such a setup. So we provide BCI which is basically a multi controller instance. And then you may have such issues with provision resource. But there is actually native Kubernetes way for that. I've already mentioned the device plugins and actually device is actually something feasible, a physical. And, for example, when I was working for my previous company, there was no Kubernetes we were actually using another platform, some retention, but the principle was the same. So we're providing a unique resource device in this case license. And when you request an agent, for example, using Kubernetes plugin, you can specify resources you want to get one provision agent, and then you can defer this license as a resource, and you will get a Kubernetes native way to manage resources. It will be the source and Kubernetes will orchestrate access between many Jenkins instances. So it's the approach you could use basically, well, in any Kubernetes based setup, whether it's a Jenkins, separated class Jenkins called BCI or whatever else this approach will work because in this case you will use Kubernetes for resource scheduling. This is something really good for licenses because license quotas are really tough to manage. And if you want to know more about licenses and Kubernetes, I can do a separate presentation about that because I spent quite some time on this one. Great. Thank you. Yeah, and speaking of that, please submit feedback because if you want to know more about such topics, submitting feedback form is the way to do that because then we know about these questions and play the backlog. Okay. So what else do we have. Yeah, any Kubernetes sharing one workspace per job. So I guess support for shared workspaces and other things. Mateusz, would you like to answer this. I believe it is possible to share workspaces you can configure it in your seed job definition and if that's not enough for you. There are some things that you can do with configuration as code to hopefully fit your needs. So that's up to you, but it's possible. Yeah, also, there is a plugin called external workspace manager. This plugin was created during Google Summer of code 2016. The first part of it was to create the APIs and to create an extension of points. And we did that use examples as an FS and other ways to share storages, which is mapping the workspaces. But there was also a full project to actually have native support for Kubernetes and for Docker. This project hasn't been implemented per se, but if someone is interested we have all the APIs so small matter of programming and you can have a Kubernetes native implementation for workspace sharing with an agent. And I guess in this case, there are also solutions based on Kubernetes resources. I'm not 100% sure, but I think it's technically possible. So for example, Tecton does so that it's, it's, it should be possible to implement it in Jenkins as well. Okay, yeah, I'll share the link to workspace manager. So feel free to try it out but yeah for disclaimer it's not fully ready for Kubernetes use cases yet. Okay. Yeah, how will the resource allocation of POTS happen? Well POTS are provisioned by using the Kubernetes plugin. So, well, first of all, you can define POTS templates there. And for example, use labels to decide which job should run where. And of course you can define a resource limits for those POTS. So yeah, and the actual allocation is handled by Kubernetes schedule. Okay, so we have two questions left. One question is, well, both questions are from Guillaume. So I guess it's a single question. So we can take it to the next part of the presentation. So that we can just discuss it just as important as the record. So there are no questions left. I would like to thank the speakers. Thank you for hosting and thanks to everyone who participated. We'll publish the recording later. If you have any questions again there is a operator chat, you will share the link along with the recording you can generate and ask any questions. And if you're interested to stay offline, we'll stop the recording and have an open discussion. Thank you so much everyone. Thank you as well. Okay, thank you.


So hey, everybody, Matteo Collina, part of the NodeJS Tech and Castilling Committee, board member of the OpenJS Foundation, co-founder of this company called, what is it, platformatic? New sweater, so thank you, Luca, for the new sweater and also author of a framework called Fastify and a bunch of other stuff. So if you're running JavaScript, if you're doing any software development, probably your software, my software is on your computer. So I'm, you know, that level of penetration. So, you know, donations are fine and useful. Thank you. Coffee works too. Hi, everyone. I'm Christian. I am working as funding engineer at Stateful and involved in the OpenJS Foundation as a contributor to the Reptaro project, which I have been maintaining for, I think, over eight years now and seeing the project grow over time and, you know, trying to build a community around it has been interesting and challenging at the same time. There we go. Okay, so I think maybe a lot of us know what community is. I'm just wondering, all of you sitting here, do you all think you belong to part of a technical community, any kind of a community? Yes. Anybody not feel like they're part of a community? Who's not involved yet? So, so we can kind of skip over that because I feel like we sort of all kind of have that grounding on what a community is. So when you all, for your projects, think about a community, what are the goals for you on building a community? Okay, I'm taking it. So they there are two kinds of essentially open source projects, probably more, but two major, two major families of open source projects. One are essentially company funded open source. So it's a way of distribution, essentially, and the company retains all the governance of the project. The other ones are the ones that adopt an open governance setting where you essentially the ownership of the project and the way the project is run is shared across multiple individuals, typically volunteers, typically spread across the community. Well, this is the typical, more of the typical vision of what we collectively think as open source, but you can totally be an open source project and not have an open governance or possibly also vice versa. So I just wanted to clarify that because a lot of the things that I'm going to talk about, especially on the concept of a community, are on the idea of an open governance project, which most of the projects that are open source foundation are. So the catch is what's the goal of having a community? Well, the problem is if you are a maintainer that creates this open source project and you release it to the world, it's the state, the fundamental problem is you want the goal for you should be to remove yourself from the project and the project to keep you running. And this is ultimately the like if the project can outlive your interest in it, then you have somewhat succeeded. Also, because there's a lot of influx of bugs, requests and things. And if you don't, if you are alone, you will burn out. And it's full of stories of burnout in open source. And then when somebody that happens, the project stagnate and the more the project stagnate, the more you burn out. And at some point, the project dies. Or if it doesn't die, it creates problems down the line for all its users that you nurture, that you care about, and then you feel horrible about it. I speak for experience. So essentially, you are at this state of more or less tension if you are alone. So if you're not alone and you're not part of a company that's funding the operations, what do you do? While you need to involve others to collaborate and work on your project? Well, and ultimately, the goal is to spread the load of maintenance. So with great powers come great responsibility in that sense. And if you're spreading the load of maintenance, you're also spreading the fact that you need to also spread the governance of the project and give you know, something away to some extent, and which is good though, because it's, you know, you want, usually if more people, more diverse people are involved in the decision making process of a project, the actual outcome is going to be better. That sense. So that's, so ultimately, the goal is for the community to some extent to take over in running your project, at least in open source. I want to add something to that. I totally agree with Mateo's view on that. Another value that a community can bring for companies adopting open source projects is security, right? You know, with company or corporate driven projects, you know, if the corporation A doesn't pay maintain a B anymore, the project is likely to die. And a lot of companies out there, they seek and look out for open source projects that are community driven, where they have some sort of security that, you know, when maintainer A disappears, there's a maintainer B that takes over. And that is a big value add that, you know, with community driven projects that comes with it. And we have other communities too. In JavaScript, we have the OpenJS Foundation communities, right? So Mateo, you're on the board and Christian, you're on cross project council, you're cross project, cross project council talk about the importance of building a community across projects. And why is that important? I think generally, we all, you know, work in the same field, we all work with JavaScript. And it's just great to have people, you know, to listen at the call up summit to people to the note community, and and see what they're gonna be up to. So I can know as a, as a user of Node.js with my framework that, okay, these changes are coming up in order to prepare my project for success in long term. Those are the changes that I need to prepare to put into the project. So the challenge in so why the CPC exists or why the OpenJS Foundation to some extent exist is there are a set of challenges that running open governance open source projects, poses to the maintainers. And most of the, a lot of those things are not really are not really in the skill set of the of the of the maintainer themselves. So because of that, group like the cross project council and the foundation and the board and all the things that touch so it exists. And it provides great services on the projects. It's, to some extent, the way I see it is is worth that needs to be done. To some extent, it's, and it's great that that is, or I didn't know that was, I think it was Sarah chips that mentioned it's the bureaucracy of open source projects or something. The bureaucracy of open source and, you know, and it's great that people are really wanting to get that bit done, because it's actually, it's actually hard. Okay, like, and from help in, you know, conflict escalation and moderation and all sorts of things are very hard. And a lot of time, it's a different skill set compared to what we usually do. So Yeah, there's all kinds of communities. I mean, I built my career on community engagement and marketing. There's a big legal community in open source. Now I have my executive direct direct director community at the Linux Foundation. So you do learn a lot from each other. And just, yeah, and yeah, you just learn and can apply a number of scenarios. So let's talk specifically about web driver and maybe fastify on how the community has what sort of what are some concrete things that you have done to build community and what are the the direct benefits that you've received from the community. So one of the things, you know, there are different times of contributors, right? The people that pass by and post issues, the people that maybe do some regular commits and contributions and then the core people. And in order to kind of like, increase contribution from all three sectors, you got to kind of like build a funnel, how can get more people in these in these buckets. And one of the things to get people interested in contributing is like, and at my, that works at my size for WebDivio, is like, provide open office hours and invite people, hey, if you want to work on a pro on an issue, particularly, I'm going to help you, you know, conquer the code base and in, you know, win with your first poor quest and contribution. I set up a chat system or like it created discord server for people to join where, you know, I can see what the pain points are for the people that I cater to and, you know, see what they want to see from the project and to understand what needs to be built. And there were various of other things that, you know, can help to build communities around the project. So for me, this was actually like part of Fastify and the decision that we I have made in creating when I started to think about starting this project where very deliberate in what were the values of that specific community. So where came from to be honest, specific planning phase or a specific looking at the status of the other projects and what what was what was the problem there and why major projects in the Node.js ecosystem had essentially close to no maintainers on on them very small teams. And how could you start to build a community where, you know, it's everybody was chipping in and collaborating to maintain, maintain the framework. So what we have done was so the first thing was, well, I'm just going to start building that this new technology, if I can convince another human being to build it with me. So this is that I started with one person, I mean, in order to do this, I need to convince another person to build it with me. Okay, this is the and from there, which is the entities of a lot of our other open source projects are born. Okay, I have an idea. I sit down and code it and build it. Now, this was if we want to build this, this is going to be way too complex, require me way too many people. I don't know many of us so Fastify is an HTTP framework for for node in order to implement this kind of stuff, you need to know the HTTP specs in and out. And it's actually a lot of plus a lot of JavaScript. There's a lot of API design, interesting problems to solve over there. So anyway, what I've done is, and then, you know, create another point come from an previous experiences where, you know, I was always tried if somebody opened an issue to answer that issue for them and solve that problem and put a fix. And instead here, I just decided, nope, I'm not doing any of that. I'm just saying, look, if you want, this is the commons. Okay, so it's your house as much as mine.   And if you have this problem, you know, it's the toilet is broken, you know, it's, it's everybody's responsibility to fix it, right? So that's the reaction has been fantastic. Okay, and enthusiastic or not, in the sense. But that's the interesting part. I was not caring at all of, at least at the early stage, in attracting the people that didn't want to be part of the active community of maintainers. So this is the most important part. You want to cater for that specific block of people that are going to sustain the project long term, especially at the early stages. So and keep growing that group of people all the time. It's expectation setting, right?  But if you create a clear expectation of like, here's a bug, you filed a bug, that's good. Here are the ways how you can fix it. People are incentivized to get back. And I tried that in my project where it was like, instead of going about myself and fix it back within, I don't know, an hour or two, I gave people exact directions. Here's the file that you need to look into. And people were field empowered, and they're like, okay, cool, I can take this on. And, you know, this saves me time and makes people feel empowered and do their own changes. So it's not just lofty words and, you know, on setting culture, like just some real specific things that people can walk away with on on building community. Yeah, I would love to see in open source in general, more incentives where maintainers go out and provide people clear direction is like, where you can contribute, where can you provide value because there's so much we talk about, you know, open source projects always about code contribution, but there's not there's like so many value streams that open source project can, you know, receive or want to receive. But often there's just a missing documentation where it's like, what do we want? What do we need? And communicate this clear to the audience and to the community so they can feel empowered and to contribute back. Let's talk about the stages of contribution. I mean, I think one of the misconceptions I hear when I speak to maybe if you don't speak at an open source conference, you speak at just a regular tech conference, people think you need to show up to a project with 10,000 lines of code and solving a huge problem that no one else has figured out. So talk to me about the stages of contributions. I think we have one, for example, of Claudia, right, he joined the new chairs community, making great conversation to, to, to the documentation system and then, you know, you know, become more experience of where the community needs help. And I think that's where everyone can just go about if they have an open source project that they like, just join the community, look in and, you know, just by even like helping other people to succeed with that tool is already value enough to, you know, start, start with that.  I think that's a, that's a great way to do that. Yeah, but I tend to see four stages in the of somebody, of people, you know, using open source. The largest group is what I call them, the lurkers, which do not, the passive consumers, or putting it in another way. You are passively consuming open source. We are all passively consuming open source in one form or another. I am a passive consumer of the Linux kernel. I'm not contributing. I'm not opening issues. I did in the past, but that's a long, a long time ago. Okay, I decided not to, it's too much work. But it's, it's a great project to contribute to if you like that kind of stuff. But the gist is, these are the passive, you know, the passive consumers. Then you have more active people that can, you know, open issues, provide reproduction, maybe every now and then on the issue tracker, you can see their name. And some of these is, these are value substages. There is the first one that just pop an issue and then disappear.  Okay. And you want to nurture them. Then you have people that start to produce code and contributions and non-code contributions too are valuable. Anyway, help the project. And ultimately, then stuff is people, oh, this guy or this gal, they've done a fantastic good job of maintaining the providing code that I want to bring them as part of the maintenance team. And then this is, to be honest, it's a funnel. Okay, you need to think of it as, to be honest, it's a funnel. And you need to think it that way and see that you want to help people. You need the end, you need people at the end of the funnel to make your project sustainable. And these needs to have value for all the actors involved for successful, for a successful open source project. So, and they're great tools that help you to create those funnels. I use a tool called orbit and they provide free services for open source software. And I see when someone is not engaging within two weeks and I go back to them and nurture them, it's like, hey, I check in with them and say, hey, what's up? Everything okay? And should we work on something together? So, they're tools that help you to create this, to build these funnels and categorize your members of your community. And what other, how else are you using data to measure your community engagement? Thinking about from your small projects to maybe some of the larger projects that node. I mean, are you regularly, is there a gut check on that? Do you notice patterns? I have to admit that I open NPM stats almost every day for the last six years. Seeing the stats going up is like, for me personally as a maintainer, exciting to see. But yeah, to come back to that, I have been playing with a lot of dashboard tools for open source in the past. And so far for whatever I've been settling with orbit, they provide you great, they give you all the list of members, they give you information about how active members are. And it's have been really helpful tool to categorize your community members and the organization behind that and see how they engage with your project, not only on the GitHub, but also on Twitter, Discord or YouTube. That has been a great tool for me. Apart from download metrics, which are very, very helpful, it's something that is even more, to some extent, it's on when growing a community you typically need to, you have a few bits that are fantastic. So when you, for example, you need to use a chat system. And with chat system, you can always, if you're not having a chat, it's a massive problem for open source projects. So you want, you really want to have some level of a chat, so that you can very quickly talk to somebody without thinking, oh, how can I find, how do I find their email address? And this is very simple thing.  And when you ever start having a nice of a little bit of a community inside the chat, you are at a point where people start helping each other. So if you look into the Fastify Discord, it's basically, essentially, self-serve. Like, we need, most of the, like, only a few members of the team are there, but most people just help each other. So, and that is super helpful, most of the time, for them. So you reach that stage where, essentially, you have, the community is helping each other. So it's not relying on you for help, which is the, to some extent, that goal of it. So it's great to see. So we've talked, you talked a little bit about different types of open source projects. They're company-led projects. There's community-led projects like Node. There's still community across both, but how would you, I mean, what are the differences from community building and participation? Yeah, I would add two more types, which is, like, the solo projects, right, the hobby projects that people have on GitHub, and the so-called monarchist open source projects, like Python, would be a good example. All these different kind of types of different incentives and different interests of how to build a community. For community- driven projects, especially, is like, they live based on the community. All their value streams come from people that just have fun contributing to the project. And, yeah, for that is particularly important, for those projects that are pretty important to build out this community that provide value to a project. When it comes to then corporate projects, you know, the values, they are not that much interested in building the community because they have the company A that is funding. And, you know, that works. And for, I think, the foundation based projects, GraphQL, as an example, they have an impact enough that there's enough interest that companies fund the project with people and money. So, when it comes down to community building, those community-led projects are most, you know, affected by that. Yeah, it's the growing, I, you know, at this point, I'm part of, I'm doing a startup. So, in open source, our main product is open source. And we are essentially doing it again, essentially building a little community around this. And you can see the difference that there is a fundamental difference and fundamental different expectation of what a community for, for a project that's essentially run by a company versus a community that is born out of, that is an open governance community. Okay, there are two fundamental, there's two fundamental difference. And in one it's, you know, you are, in one, you need to help the people using your stuff. Okay, because you are creating the community, you're going in the community to, for them to make it easy for them to use your product and so on. On the other end, you're doing it to survive. So, it's kind of different. And it's both ways, because you don't want to, if you're building a corporate community around a product, okay, you are, you're building it for, and, but you don't want to transform them into mindless fun boys. Okay, if I, you know, somebody's laughing because I, this is probably a reference to a few things that are out there, but it's, you know, there is, I'm pretty much skeptical when somebody's just super becomes a zealot of certain, certain way of seeing the world. So, and I still, you still want to be in a place where you value critics, you value, you get there, they can still be part of the decision-making project, the process of, of the product, but it's, it's a slightly different dynamic, as I said. Interesting. Okay, so if someone out here is, you know, they're, they're here at the conference, and they want to become part of a different community, this community seems great. What should they do? They would go out and see projects that they find interesting, and I think it's all, you know, having engagement comes out of personal interest, mostly, and every community often is welcome enough to just, you know, have people looking around, and then provide them ways to contribute to their community, and like, I can just re-emphasize what I mentioned before, it's like, I try to, everyone who joins the Discord channel, I'm like, I want to check in with them, and be like, hey, where do you want to contribute, where do you want to, what are your projects and your fields that you're interested in, and I hope that every project has something like that, and if not there's, you know, as someone who joins the community, it's definitely not wrong to create an issue and be like, hey, I have these skills, how can I help you, you know, with the, in the project? They can do that, yeah. Awesome, I love that idea. It's amazing. Works very well, yeah, for, for Fastify is mostly on the, on the issues, so, you know, it's, at this point, it's, it's, it's very, it is becoming more, a little bit of a meme to some extent, it's Matteo asking, somebody asking people, hey, you know, it's a good, this is, thanks for reporting, can you please send a poor request and remember to add unit tests, and this is the, it can be a meme at this point, you know, it's been, been asking these people all the time, all over the places, and, you know, it's, well, you know, the answer there is either you're a member of this community or you're not, and it's, you know, it's, it's what it is, so a few people have said, no, I'm not using your stuff anymore, and this is the answer that I got a few times, and I said, oh, good, that's my, the answer is, yeah, great, it's, it's one last thing to, to support at least, that's, you know, you don't want to engage here, so you're, you're, you're basically self-selecting at this point. One concrete example that I've recently experienced is, Augustine, he chipped in the call-up summit, he created an issue and said, like, I would like to see the WebDive ODocs translated in French, I'm like, okay, let me revive my engagement in, like, with crowd-in and connected docosarrows with crowd-in, and I hired people on Fiverr to kickstart the translation and translated, like, the documentation into Spain and Hindi, cost me 800 bucks, I translated German myself, and then I released about the website in three different translations, and that has kickstarted, so many other people will be like, wait, we can help translating the website, it's like, that's awesome, and then people now, we have this issue for translating the documentation, people go there and be like, hey, I can translate the website in, I don't know, Tamil or other languages, and I'm open in the language for them, and they just go ahead and translate it, that's like, I find this awesome, and one of the examples where it's like, someone comes there and be like, hey, I can provide translations, and, you know, you help them be successful contributing to the community. Do we have any questions from the floor? Claudio? Yeah, the first question that I have is, it's related to mental health of maintainers across, for example, internet can be very non-forgiving, right, at times, how you deal or how you should at least have dealt with the unfortunate moments where the community at large isn't really nice with the maintainers, how do you keep the cycle of them feeling motivated to keep contributing to open source even when the community is at their necks, you know? I have a long answer here, so if you want, I'm sorry. I can just answer from my point short, when I get fed up or when I feel burnt out, and I see there's so many things coming at me, I see that at myself by not being very kind in my answers, now we're going to become short, and I see that and then I step out for a day or two, be like, that's enough, and then I come back and see things with different eyes, and especially when, you know, other people's posting issues or answering to your issues don't answer with the nice set of respect, I just step out and come back and then the world sometimes looks different, and then move on. I had some pretty bad burnout, experiences where I stopped maintaining projects, as you know there is a few of mine, I have a few graveyard projects or putting projects in the graveyard that they left out there, and look a few of them are still, I don't know, a million downloads per week or half a million downloads per week or something that makes, you know, why the heck you abandon this or whatever, comes from multiple set of the toll comes down to what I call there is an imbalance between the number of the community expectations and the number of maintainers to support those community expectations, okay, so it comes from this imbalance, and you need a certain, you need a ratio, but this is different from each project, but you need to have a level of representation of these amount of maintainers are needed to, for this large, at large community to exist, and if things are not meeting those expectations, if things are not matching, you got this situation of burnout or frustration, and it can be good at times, so you can, you can have a peak of things, a peak of influx of new members of the community of users of your, of your open source project, which makes it enlarge the community, so makes more bugs, more things, then it goes down, because for some reason you cannot service them, and they just, no, I'm fed up, I'm going back using Python, okay, and not using Node anymore, okay, and Node does a lot of surveys and collab summits to prioritize. Yeah, pretty much, so again it's, so to be, this is a good example, thank you Robin, so essentially it was a few years back, there was this perception in the community that Node was stagnating, and Node was not following what they, where the need of, of the community, okay, and I remember being in, in a collab summit between 2017, 2018, something like that, where I, I laid out, I don't know if you were in the room with another time, or at this point, but I laid out a plan, or an aspirational plan to ship fetch in Node, like the thing that everybody was requesting in Node.js was, let's say we need fetch in Node, and, you know, I was discussing it, and it was, I laid out this, it's still online by the way of the presentation, and I laid this plan to ship, to ship fetch in, in Node, and it only took, I don't know, maybe four years to get there, it's, you know, I said, with a lot of meme, we made fetch happen, okay, it's, you know, but it's, because those things take time to move, okay, from time to time, it's not just as simple as a lot of people would, would think in the community to the two, make something like that complex to actually happen, but eventually we started back listening to what the community wanted to say, was saying for to us, and then we started adding those prioritizing, we started a group called the next 10 years group to decide of the technical priorities of the project, and some of those are still, were, were those things, and some we shipped, like every single body, every single one, the top most needed thing that everybody's telling to these days is, oh, you should get better typescript or more tight typescript integration. Now, this is a massive amount of debate in the community on how to do that, and a lot of very interesting discussion happening all the time, but this is essentially part of, of the process. But again, listen to your community, your community more or less will ask for what they need, and from some of that group, there are probably also potential people that can help you do those implementations. Otherwise, it won't be, it won't happen. So yeah, that's, I hope I answered the question. I think so, priorities. We have one minute. All right, Sameer. Thank you, Robin. Great topic. My name is Sameer Johola. I'm the executive director of Agstack, which is a project of the Linux Foundation AG in agriculture. My question to you is, how do you, what tools and processes have you used to found to be useful to engage the community to solicit the topics themselves, the questions that are worthy of a community, so that it's not just the loudest voice, one or two people who kind of are interested in coordinating, but they sort of drawn out the quiet voices. As an example of that, in our community, we have a really interesting problem. The tech people don't really understand agriculture, and the agriculture people don't really understand tech. And so we really, when we ask the tech people, what are the questions they want to answer, the results are completely different and irrelevant to the agriculture people we ask them, and they ask, give questions about the topics that are totally, you know, go over the head of the tech people. So this may not be a direct translation to what you guys have been doing, but how have you found to be successful convening of questions? What are the questions to ask the community about, so that not the loudest voice always wins? That's a good question. Trying to translate this to my use case or enterprise demands on projects. So from my, so typically what the way I have approached this and helped in that decision making is first thing you need to, this is Brutal, try to gag the loud voices. This is as bad as it sounds, but you need to create a place where the quiet people can talk. And from time to time it means pushing out somebody that is very loud, very opinionated, and does not leave other space to express themselves. Or most of the time they don't even realize that they are doing it, and you just need to tell them, oh please let the other people speak, okay, or and so on. They are considered, especially in open source, that is, and it's a phenomenon that I've seen multiple times happen in the North community, is people that engage in wall of text to prove their point. Okay, and it's, somebody knows what I'm talking about. So if you want to stall an issue, you know, progress in open source, you just keep writing long and long pages of text to, to block it essentially.  You know, this is what, this is, and at some point, if you are on the leadership of the project, you might need to intervene to essentially stop that. And your community needs to be able to stop that specific part. Now, if you can stop that and allow everybody to contribute, this is the first step to answer that, to go into that direction. Okay. On how do you get to ask the question, you, the way you, the way you're describing it, you have a community that is true-sided and those communities that are true-sided are, are created around a few special individuals that are part of both worlds. And identify those special individuals, the people that are able to, to keep one fit in one, in one world and, and in another and they can help you create those bridges. And those are the people, and typically they're very quiet too, most of the time. So, but you need to get them to speak and that will give you the answer. And otherwise a project like, like yours, there are already because the project otherwise will not exist without them. It's just identify them and let them. What are the questions that, you know, let them speak essentially. So, I hope I, I give. Yeah, I, I second that. It seems that your community is kind of like separated with two groups that have different desires and pain points. So, one way could be that you cater your community engagement to these different groups individually and filter out the responses and be, then be able to connect the white people with each other based on the responses that you get. Like it's, I think it's hard to cater a single value stream of content to both when they have both different pain points and desires right. So, separate them and like look at them from a new individual perspective and then connect the dots based on the responses that you get from the community. Well, thanks everyone. I just want to close for folks who are watching this online. There's a number of ways to get involved in the community. Hi, Mom. We have a lot of us publish our calendars of our meetings on our websites. We have one at OpenJS, the Node Project Web Driver. You have it as well. A lot of the calendars are on GitHub. So, yeah, find us. We also have YouTube channels. We stream most of our meetings live on YouTube. So, if you want to see what it's like you can go check them out and join us. Thanks. Thank you all. Thank you.


So we've been watching the impeachment proceedings play out now for multiple days and I'll admit this is absolutely exhausting, albeit necessary, because I don't want to live in a two-tier justice system. Like I believe that there really is no justice in the United States and we do effectively live in a two-tier justice system already where the rich get away with committing crimes and poor people get locked up for doing those same types of crimes that rich people get away with. So I think that we need to hold everyone to the same standard and, you know, we can't give people a pass because it's politically expedient. So even if, you know, I would much rather be watching, you know, people debate Medicare for All on the Senate floor, this is necessary. This is something that we have to do, although I do understand the concerns of lefties who feel as if this is going to be a distraction with regard to the media and Democrats will just use this as an excuse to not focus on policy substance. I get it, right, because Democrats by and large are incompetent. But part of the reason why I really support impeachment is because one of my main criticisms of Democrats is that they are too weak in the face of Republican opposition. And when they're finally like fighting for the first time in what seems to be forever, it makes no sense for us to say, actually, don't fight now. When you're finally fighting, actually, don't forget everything I told you. It doesn't make sense, right? And also all the concerns that we were worried about or that some people were worried about with regard to this possibly being a distraction. I don't really think that that's played out. So AOC talked about this a little bit in an interview with Chris Hayes on MSNBC. And she did a brilliant job at really debunking all of these myths about impeachment. And she explained why this is absolutely crucial if we want to live in, you know, a country where we respect the Constitution and the rule of law, like we don't have a choice. We have to hold elites accountable. And also the fears that everyone had about this being a distraction. That really isn't bearing out in reality. So this is what she had to say. You're someone who I think you came to Congress with a very strong vision and agenda. You were clear about that when you ran your primary that you won. And I think there's an interesting sort of debate about this impeachment and about the president that has to do with, you know, is this the best use of legislative time? You've seen some Republicans saying it's not. As someone who has an agenda that you want to pass, right? That you want to see happen. How do you view impeachment in that context? Well, I'm not very concerned about it because we're able to legislate while this is happening. Just yesterday, I introduced our first piece of Green New Deal legislation, which was around public housing and decarbonizing our entire public housing stock in the United States. And so it's not coming at the cost of legislating. Some may say that mass media may cover our proposals a little bit less, but they don't do a great job of that in the first place. So it's, I love you, I love you, but I don't think I'm going to get like a decarbonization like 8 p.m. time slot. So it's okay. I think we're legislating, we're working for people and we're holding the president accountable and it's all possible. And if we don't hold this accountable, then we really erode rule of law in the United States of America. And really what makes America different, when people say, I want to do business here, I want to, I want to write books here, I want to take my family here, I want to raise and be around American ideals, a lot of it has to do with the reliability that people, the right people will be held to account, that there are consequences for doing wrong, for hurting people. And also that this is a fair country where everyone is treated equally. But isn't that as often honored in the breach as not? I mean, one of the things that I think helps him is there's so much cynicism about that exact thing. Well, that's exactly right, isn't it? Because there is a lot of corruption in our society.  And there are a lot of systemic threats.  And at its core, the most sacred document in our society is the Constitution of the United States. Everything else is very easily amenable, but this is not. And once we erode the general respect for the Constitution, then we essentially erode respect for the United States of America. And that's what this president has done. So I mean, everything she said there is incredibly important. They're still getting things done. Just yesterday, she said, I introduced our first piece of Green New Deal legislation, which was around public housing and decarbonizing our entire public housing stock in the United States. Yeah, we can walk and chew gum at the same time. I never really understood this argument and I didn't find it persuasive. That impeachment would take up so much time that it would be a distraction. First of all, Donald Trump is president and Republicans still control the Senate. So it's not like we're going to pass very many policies anyway. Not really going to happen. So to say that this is going to distract from talking about policy, not really because the same thing is going to happen that has always been the case.  That's not going to change because of impeachment. So she makes a really great point about that. She also says some say mass media will cover our proposals a little bit less, but they don't do a great job with that in the first place. Exactly. The media is never going to all of a sudden focus on policy unless there is any type of sensationalist value that they're able to, you know, extract from it, right? So they're not going to talk about Medicare for all debates on the House floor unless there's drama. It's an exchange that gets a little bit too heated between people, you know, in Congress debating it. They focus on sensationalism because sensationalism is what yields ratings and ratings is what yields profits because that attracts advertisers. So it's not going to be like, oh, well, all of a sudden they're going to start doing a bad job. They're already doing a bad job, right? So to say they're just going to be hyper focused on, you know, impeachment in the same way that they were focused on Russiagate, look, the media is going to focus on what they believe will get them ratings. And nine times out of 10, that's not what they should be talking about. The substance always gets forgotten about and swept under the rug in favor of, you know, horse race politics and political drama. That's the way that mainstream media operates. So that's not going to change because of impeachment. And I don't find that argument persuasive as well. Now finally, what I think is the most persuasive argument that she makes in defense of impeachment and in defense of the impeachment proceedings that we've been seeing is this is about the principle. Why don't people understand that? If we don't hold this accountable, then we really erode rule of law in the United States of America.  Right. Just because we let elites in the past get away with committing crimes. Just because Donald Trump may get away with committing other crimes doesn't mean that we shouldn't at least try to hold him accountable for this. Now I do absolutely sympathize with people who believe that we should broaden the scope of the impeachment inquiry. It should be focused on the emoluments clause, the hush money payments and the general corruption. Like, of course we should focus on all of that. So I don't agree with the way that Nancy Pelosi is proceeding with this. However, that doesn't necessarily mean that I don't think that the Ukraine call in and of itself is an impeachable offense, because of course this is an abuse of power. You can't do this.  They just act impulsively. And even though sometimes that hurts them, overall they've been more effective at winning elections and getting the message out there because they never back down. So we need to make sure that we reinforce the message that Democrats have got to be strong and we actually need to applaud them when they finally listen to us and they try to hold Trump accountable. I mean, we've been pushing Nancy Pelosi for this impeachment inquiry now for more than a year and only now she's finally doing it and, you know, she could broaden the scope of it. But don't tell them to back away when they're finally trying to exert a little bit of pressure on Republicans and Donald Trump because we want them to fight. Democratic Party weakness before is what led to their defeat, right? So being strong in the face of Republican Party criticism, I think that that does matter. So, you know, all of these fears that I think lefties had, some of which are legitimate, some of which I don't disagree with, I think that it is important that we address them. Although, you know, what AOC said here, I think it absolutely makes sense. One of them are persuasive enough to where people on the left should be discouraging this impeachment process from taking place. I mean, it's already basically proving that Donald Trump did in fact abuse power. So he should be impeached for that. When presidents break the law, we should hold their feet to the fire. This should be something that isn't controversial. You know, the right is united in fighting against impeachment. So the left overall should be united in pressing for impeachment. So yeah, I just, I don't understand why some people are against impeachment on the left. I think there are legitimate concerns. I think you can argue about the way that impeachment is being conducted. I think Kyle Kalinsky makes a pretty solid argument as to why Trump should be impeached for a different reason as opposed to this Ukrainian call. But with that being said, to just be against impeachment overall. It doesn't make sense to me as lefties. We should always push for justice. And that means we fight against the two tier justice system that we live in and we make sure that elites like Donald Trump don't get away with crimes that they commit brazenly so.


A promising technology for removing atmospheric CO2 is called geological carbon storage and in this technology we take CO2 captured from coal-fired power plants and inject it into the subsurface as a storage option. And so the goal of our project has been to find ways to engineer these systems so that we can store the CO2 more effectively and more safely for longer time periods. So this is pure magnesium carbonate that formed naturally in the subsurface. So all of these rocks came from a magnesium carbonate mine called Red Mountain which is located about 70 miles from here and it is one of the largest examples of a natural deposit of magnesium carbonate and it was mined economically for about 50 years. And so one idea would be to inject the CO2 into these different rock formations and try to convert it into a carbonate instead of store it as a fluid. And in this way we would permanently lock up the CO2 as a stable mineral in the subsurface. And so we've been looking at the structures and the veining to try and understand how the rocks were transformed into magnesium carbonate.


So, what you are referring to as love is a certain moment of pleasantness. If love is true, it will happen without an object. Every moment of your life you must know that pleasantness. You must constantly fall in love with everything and nothing. It all depends what is your idea of love, first of all. If you fell in love with one woman, one bird, one hand, one tree and something, then your love is not driven by anything, you're just falling in love. If continuously you're falling in love only with women, that means your love is driven by your hormones, you better understand that. So what is driven by something, you want to label it as love, it is fine. You are not saying there's no love at all, there is. But what is true and what is not true is not something that you decide because of this, you know, because of what happened or what did not happen. It is a question of how you felt. Whether it ended up in marriage, then you realize it was not love or it did not end that way and you thought it was true love which was never realized, it's up to you how you want to handle it. But essentially what love means is if your physical body becomes very pleasant, generally we call this health and pleasure. If your mind becomes very pleasant, we call this peace and joy. If your emotions become very pleasant, we call this love and compassion. If your life energies become very pleasant, we call this blissfulness and ecstasy. So what you are referring to as love is a certain moment of pleasantness. Now you believe somebody caused it in you, maybe somebody did trigger it in you, somebody did stimulate that in you or inspire that in you. But essentially it is your emotions turning very pleasant which you call as love. So if love is true, it will happen without an object. If it's really genuine true love, what it means is you have made your emotions very pleasant. Whether a man is here or a woman is here or a tree is here, a ant is here, elephant is here is not the point. You've just made your emotions pleasant. Everything that you behold, you have a certain pleasantness towards it, that is love. Now this continuously one, two, three, four, five, all are women, it's driven by hormones. Nothing I'm not saying is anything wrong with it, but you must understand the limitation of it. So if it ends up in a permanent bond in the form of marriage or whatever, then you think it's successful. If it did not, you think it's a failure, it need not be so. I would be happy if you fell in love with the three billion woman and don't leave the other three billion men also. If you fell in love with three billion woman, it wouldn't be hormonal, it would be just love. I'm in love with all the women in the world and the men also, the ants also and the very soil that I walk upon, the very tree under which we are sitting right now. Wherever you behold, you have pleasant emotion towards that, it's love. And love is not about somebody, it's about you being pleasant, truly pleasant and this is right, wrong, this, that is not the thing. It is an intelligent way to exist. There is sufficient unpleasantness in the world. However carefully you walk, you will step into something or the other. When this is so, you don't have to manufacture your own unpleasantness within you.  Every moment of your life you must know that pleasantness. You must constantly fall in love with everything and nothing. Women also included, what's the problem with them? Because you cannot tie a bond with three billion woman, you choose one. One is convenient, if you marry five, then you will find it's very inconvenient, yes? One is convenient and above all the reason why you tie a bond with somebody is, at least with one person in your whole life, you want somebody with whom there is no holds barred. If you approach a new person, always there is a little bit of, you know, there's a little bit of war and games happening. War games are happening with a new person. With one person you want to relax, where you don't have to play games. At least with one person you want to be in total relaxation, that's the idea. Why you want at least one person in that is, still you have not reached a point where you are simply relaxed by yourself, with life itself. Because you are unable to relax with the whole manifestation of life around you, the million manifestations of life around you totally, at least with one person you want to find that, that's the idea of finding a bond or building a bond with that. So people who look beyond that are people, they don't value relaxing with one person because now they are able to relax with just about anything. They are totally at ease with everything in existence. Now they don't find the need to make a bond where only at one place some relaxation happens. Rest of the place you are war-like, always trying to protect yourself.  So the idea of forming a unit or family and the larger identities of community, nation and whatever else is simply to find a space where you can keep your defense mechanism down. Just be in a certain sense of ease because only in that ease the other possibility you find expression, otherwise you will be just like an animal constantly trying to survive, preserve yourself. So love is seen as a possibility to go beyond that instinct of self-preservation where you are not trying to preserve yourself, you are trying to offer yourself to something or something. For that you don't need a person, you can just do it simply with life.