This is Google season of docs office hours with the Jenkins project. Thanks for being here and let's look at our agenda. So what we've got today is a helm chart and Mark, he's told us he won't be able to attend. Torsten is here. And let's see, were there any other topics you wanted to put on the agenda, Zina, before we go right into the helm session with Torsten? Yeah, the pull request. Those comments, Christine raised about, hold on, I just want to... Sorry, just had a cough. So the comments... Yeah, about the production, okay, the points where I noted that... So let's look at that pull request and see... I'm sorry that I was coughing and couldn't hear while I was doing it. So it was, you were Kristen's comments here or... Oh, yes. Okay, let's make that text bigger. So this is something I would like us to discuss because we really need to understand what she's pointing out here and if I'll need to make changes to what we currently have before we merge. So she mentioned, she suggested a different section to highlight this. So I'm not sure... Hello, Kristen. Is she on the call? I don't recall if she's on the call or not. She is not, not yet. So I think it's okay that you've described a way to get working here and that for me is sufficient and good enough for now and good enough to justify merging. I think her point is that there are probably nuances and important pieces of information in a production environment that we may need in the future to say, hey, in production, you would not do this, you would do that. And that's, I think, for instance, certainly my mini-cube does not have a really, a high reliability persistent storage behind it. It's just got my desktop computer. Yes. Okay, because my understanding was, I wanted to be sure what she meant was not moving this notice because I think it's like in two or three places in the dock, in the installing Jenkins, we have something like this. So I just wanted to be sure she didn't mean moving this notice into a different section. And if that was what she's suggesting then, well, I think it would be better to highlight them in different places because from experience, I know some people might go through the whole dock and actually following the step without even actually checking whether this is for tests or this is for production. But at least notice like this will help people understand that whatever it is that they're doing in the documentation should not be tried on production environment. And I think based on her sentence here where she says, I think it's great to have these suggestions for other sections, I think she is agreeing wholeheartedly with your assertion that the note really belongs exactly where you put it. So I think you're on the right path for what she's describing. Okay, thank you. Very good. Any other topics there on the pull request? I expect, so Mark is taking a day off today. I'll go ahead. Okay, so I plan to review it and likely we'll merge it today since I believe there have already been other reviews on it that have approved it. Oh, we've got comments from Kristen. So after I've reviewed it, I suspect I'll merge it if there's nothing that I consider seriously blocking. I assume Zina will be okay that if I find some minor thing I can just go ahead and submit a change to your pull request. Yeah, that's fine. Anything else on the pull request? No, nothing else. And Kristen had recommended the second pull request, the structural one, or it was a skeleton framework, right? And I think that one is also ready to merge or near ready to merge. Or is that the one that we're already talking about we've merged your first already? No, this is not it, this is the second one. Also needs review and merge. Great, all right. Anything else there before we start our home session with Torsten? No, nothing else. Okay, so Torsten, I'm going to stop sharing and let you share. Sure, no, no problem. Oh, that's the wrong one. Share my screen is better. So you can see my slides? Yes. So this was also meant so we could use it as a Jenkins Meetup, so excuse the nice styling and so on. So originally this one was a stable Jenkins chart and it was just recently migrated to Jenkins, Jenkins or Jenkins CI org. About me, yes, I'm just a developer and having fun doing some stuff. About the meeting today, I would love to give a quick overview of the chart, the agent configuration, dive a little bit into understanding agent permissions, how the stuff works, how one can configure credentials in Jenkins on Kubernetes. A little bit about the configuration as code as this is quite an important part of the chart as we moved all the configuration from XML to that one. And the last section is about chart improvements. Zainab, the session is mainly for you, interrupt me whenever you have a question or don't understand something or you can also say, hey, it's too slow, just skip over things. It's just give me feedback and I will try to adjust. Okay, thank you. So chart overview, what does it work? I mean, Jenkins, typical architecture, you've got one Jenkins controller and Jenkins agents. Best practice is that you only run builds on agents. Typically, I mean, you would love to avoid running any builds on the Jenkins controller itself because then you can read the local disk and so on. And that also scales way better if these agents are can be created on demand. On this chart here, I pointed out that typically the users connect to port 8080 or that's a standard port with Jenkins exposers. And there's another port like 50,000 where the agents typically connect to. I think nowadays it's also possible that the agents connect via web sockets to the 8080 port but so far the HelmChart just uses these agent port 50,000. And a word on agents, agents can be either static or be created on demand. So static agents with what I mean with that, it's more like you create a VM, a Windows node or so and run then Jenkins agent, that's a small JAR file and connect it to your Jenkins controller. And with that, it's registered in Jenkins and you can use it to schedule builds. Or they can be, the agents can be created on demands multiple plugins for that one. I mean, there are plugins to create agents automatically on AWS or on GCP by just using virtual machines. And there's also a plugin which spawns agents on Kubernetes and that's the one which we are going to use here in the HelmChart. So that basically Jenkins itself runs on Kubernetes but also the agents run on Kubernetes. Short history of the chart. I mean, it basically just deploys a controller and allows you to spawn agents on Kubernetes and therefore it uses the Jenkins Kubernetes plugin. Has quite some age, I would say. So the first commit I looked it up is from September, 2016 and since then it has more than 300 contributions and actually since it moved to the Jenkins GitHub org it also got some more attention in terms of issues and other things. So what do you get? Jenkins runs on Kubernetes and the Kubernetes plugin can be used to schedule agents. Also by default, the Jenkins configuration as code plugin is used so that you can configure Jenkins just with YAML files. And the real idea behind this that you do all the configuration just via code and stay away from fiddling in the UI and making some changes over there because I mean, you can do obviously you can do backups of those but if you also version control your configuration changes then it's easier to keep track of these changes. And another thing which the chart gives you is the reload of the configuration so that you don't need to restart Jenkins just because the configuration changes there. And by default the chart also creates persistent volume automatically for you. So that basically if you restart Jenkins whatever is on there as a configuration or as installed plugins also stay there for Jenkins. So Tarston, just to be sure I assume that since I feel also ignorant I'm going to interrupt you as well for questions. Is that okay? Okay, so when it says your persistent storage for Jenkins home that means that if the thing that's running Jenkins in Kubernetes for me dies I don't lose all the state that is the Jenkins the Jenkins installation will just come back up and be reattached somehow to that. Exactly. So the volume will be just reattached to the Jenkins controller and Jenkins will continue to run. Great, thank you. Thanks for the clarity. This is how the setup looks like on Kubernetes. Typically you have an ingress controlled I mean that's optional in the chart then you've got two services. One is a service for Jenkins itself and the other one is for the agents. So you see the difference is one points to port 8080 and the other one points to port 50,000 but both point basically to the same pot where Jenkins is running on. It's a singled instance just because Jenkins doesn't really support high availability. So you've got one Jenkins controller. This one is at the moment of deployment. We plan to change it into a stateful set and that PR is already merged but it will be a version three OO release of the Jenkins chart as it's a breaking change. Yeah, you see that the port itself contains out of three containers. The middle one is the most important because that's Jenkins itself. Then there's one init container. This one basically is used to download all the plug-ins and then copies this plug-in to these persistent volumes. So here you see the persistent volumes and basically also the init container and the Jenkins controller itself have access to that one. And the third one is a Sidecar container which means it's running together with Jenkins all the time. And this one basically watches for config maps. So all these jcars configurations are stored in config maps and this one simply watches for changes and whenever there's a change it updates this configuration on the persistent disk and can say, please reload that one. And what you also get here is a service account for that port or this port is using a service account. And with that one you get roll bindings two different ones. One is responsible for reloading configuration files and the other one is responsible so that you can actually schedule agents. And then you have like one too many ports just normal manages ports which basically the agents where the build jobs are run on and we'll dive a little bit into more details in the upcoming sections. So with regard to the agents those can be not just Linux agents but Windows agents. There is facility to do more than just Linux or? I think at the moment it's limited to Linux because these are ports in Kubernetes. And I think most docker images are just Linux images. I mean, if there would be a Windows image you could also use that one. That's why I'm looking also forward to add support in the chart for the web socket connections because with that way if you had for example the agents could connect just via web sockets to the port 8080 then you could connect to that Jenkins which is running on Kubernetes agents from Windows VM which are running outside of Kubernetes. Thank you, thanks for the clarity. Thank you very much. So how to install the chart? Yeah, what do you need? You need a Kubernetes cluster, you need Helm and these are the commands to install it. So I have repo at Jenkins charts, Jenkins IO and do a repo update so that it downloads updates of repository and afterwards just do an install. First one is the name of the release and the second one is telling you that you want to install the Jenkins chart. The two things which are gray here is that's optional. I did that locally in advance. So basically I created two namespaces, one for Jenkins and one for agents. So that we have the possibility later on to schedule agents in a separate namespace you will see that in one of the demos. And with that, I would say it's demo time. So let's go ahead and start Jenkins just to... On the question, I have a question about the chart, the Jenkins chart, and I think we also made a comment on who was the difference between the Jenkins and Jenkins. You mean between this one? If you name this one Jenkins or Jenkins CI, there's absolutely no difference. You could do both. So the difference is just if you do an help repo list, you can basically at different repositories and each repository has its own name. And the first parameter is just the name you give that one locally. So you couldn't use there whatever you want. And at the beginning when I set up this chart, we use these Jenkins CI and then someone recommended to just stick to Jenkins. And therefore this one was changed in the chart documentation. That's why I added it in your documentation. Thank you. So what happened is... Sorry, one more question for me. So back to that earlier slide, is the Jenkins name that is used there in the repo add the left-hand side of the argument to Helm install then? Or the right-hand side or the left-hand side? This is the left-hand side here. Excellent. So the left-hand side here is the same as the repo there. So basically I could put in there a repo and here also a repo. And the other one is the name of the chart. Thank you. And what you could do in an addition is just specify a version and say, I'd like to have a specific version of that Helm chart. For example, if you stick to one, if you don't provide it, Helm will just use the latest version. And the version number there is the version number of the Helm chart. Not exactly. And then the version number of Jenkins is represented inside the Helm chart. Exactly. So what I have here locally is just a Venetus cluster. So I do cube control get pods. That's just the abbreviation which I used here. And let's go ahead and do a Helm install. What was it? Oops, I did a copy and paste. And let's update this one to Jenkins. With these, you'll see it creates a pod. My release dash Jenkins. My release is just because I named my, gave the same release the name just my release. So you can put in whatever name you want there. You could even install multiple Jenkins on the namespaces. I mean, it doesn't make much sense, but theoretically it's possible. And the output here tells you what you can do to get your admin password. So by default, the chart just generates a random password. And you'll find that one in secret. And this is my password. And what I can do now is to a cube control port forward. Instead of using the pod here, you can also do a pod forward to a service. And this, the pod name might change. If you restart, I'm just using the service. So I'm port forwarding the pod 8080 from the Jenkins, service my release Jenkins to my local machine. I suppose the pod is not ready. Yes, it's initializing. So cube cuddle locks says that you are interested in the locks of that pod. And if you simply type it, you will see that there are multiple containers. One is the Jenkins container. Jenkins SC config is for the sidecar configuration reload and these init container copy default config. And if you don't specify a container, you'll just get the output there. So now I say, I want to have the locks of the copy default config, so the init container so that you can see what it does. And at the moment, what it's doing is just downloading the different Kubernetes, different Jenkins plugins, which are specified in the chart. And therefore it's using these install plugins as a script. So that's also something which should be changed to use the Jenkins plugin CLI. So you see, I downloaded all these files, copied it to the shared volume. And with that one, the init container is there and now the first of my two ports is running. So we could give it another try with the port forwarding that seems to work. So localhost 8080 and where did we have these nice password there? So username is admin and the password which you typed in there, sign in, that's my Jenkins. So this configuration is intentionally structured for configuration as code. You don't, the chart does not take the user through the typical setup wizard. Exactly. So at the moment, the setup wizard is disabled and you do all the configuration as code. I mean, the intention behind is that it allows you to set up Jenkins in a completely automated way. I mean, in the past I used to manage like nearly 200 Jenkins installations and you don't want to do that one manually anymore. Great, thank you. Yeah, so no need for the user, the consumer of the Helm chart to learn how to disable the setup wizard. Great. It's disabled automatically. I mean, we are thinking, I mean, it might be an option in the future to enable the setup wizard again and just configure the admin password via that. Let's see how it goes. So at the moment, what you can do is let's just create a normal job, call it demo, make it a pipeline job. And I mean, this is Jenkins, just as you would expect and I'm using the Hello World pipeline on agent any. So it takes whatever agent you'd like to use, please use it. I mean, otherwise you could specify agents based on labels, so say agents on Windows or whatever attributes your agents basically have. You see that build job kicks off and you've seen before that we have just one build port running, just one port running, that's the Jenkins master. If I do it again or do a watch on that one, now you see here is a container creating. This is the default Qtay or whatever container. And this is the Kubernetes plugin, which now creates a container for this one, for this port, connects this one to the Jenkins controller and runs a build job on this one. So here you see also the new build executor, which is a new job, which is currently starting up and connecting to Jenkins. And once that's done, waiting for next available executor, then you see this nice Hello World message. And here you see in the output, the port which was created, you see also the request limits and so on. So everything you would expect from the Kubernetes port and here's our Hello World message from the pipeline. And after that, our build executor is gone again. So if we just do Qtay or getPots, you see that the other port is gone. So that basically, in the default setup, these agents are just started on demand and when they are not needed an email, they are deleted. So now, is there a reason we're not, you had mentioned a sidecar container earlier, would that be visible in the list of pods? Or that's, I would be listening. No, the sidecar container belongs to these pods. So you see this. It's already inside that pod. This pod consists out of two containers, two images, two containers in this case. One is a Jenkins controller and the other one is a sidecar. So these two are running all the time. And in contrast, the init container basically is started in advance and once the init container is finished, then Kubernetes will start these two containers from Jenkins itself. Makes sense? It does, thank you. Thanks very much. So one thing which I find quite helpful is, I love to show how to configure a log recorder because sometimes people are in a situation that Jenkins is running on Kubernetes. Someone else is managing it for them and they have no clue why stuff is not working. And I mean, Jenkins provides these quite useful logs and you can just create a new log and configure our logger here. And the org sees interest Jenkins plug in Kubernetes. That's also documented on the plugin website. So nothing to worry about. You add this as a logger, say save and then it's going to log everything, the Kubernetes plug in does. And that's also quite handy. So if you would start the build job once again, then we can see in our log that it says in provisioning at the moment, it says excess workload after pending Kubernetes agents one. So it's going to use a template for label null. It's just using the default templates. It's built up the connection to Kubernetes. And then it's going to start another agent. And you can see here the whole Portsback it does. It creates this Ports. You see the Portsback once again. This is just normal Kubernetes stuff. And the containers are running and then basically it's waiting until that container connects to Jenkins and it's going to delete it. And for debugging purpose, I would say that's quite interesting. You see at the end it's disconnected. It's before it connected to Kubernetes once again and basically terminated the instance. And this termination of the instance basically just means that the plug-in uses its permission to delete that part. I would not leave that running all the time in production. I mean, obviously if you have many build jobs, the logs get quite lengthy, but if you are in a debugging session and can want to isolate something and can make sure that just one build job is running or so that's quite handy. What else? Ah, yeah. I wanted to show you where this stuff is configured. So at the moment you see you go to manage Jenkins, manage nodes and clouds. And this is the Jenkins controller itself formerly called master. And then here you've got a config section says configure clouds. And this is where these dynamic agents are configured. And there you see one Kubernetes cloud called Kubernetes. I mean, you could even configure multiple of those. So you could have, I mean, in this case, it's the chart just uses the Kubernetes cluster where Jenkins is running. And you could also connect it to Kubernetes clusters outside. So even schedule agents on different Kubernetes clusters if you wanted to. Then for these clouds, there's the details. So this one tells in the URL of the Kubernetes, the namespace where the stuff is in the URL of the Jenkins server. So that you have it in your build jobs, the Jenkins tunnel is tells the agents how to connect to this Jenkins controller. And obviously this one here just works inside the Kubernetes cluster. So outside this connection URL wouldn't work. And you see here POT templates. And these POT templates are basically agent templates for Kubernetes. So you saw that we have one default template and there could be more of them. So you could configure them with different images, different whatsoever, different requests and limits and so on. And so POT templates, do they typically describe the size of the thing that will be running there in terms of memory or processor requests or is it more about software configuration or both? Both. So it describes every attribute you can configure for a normal Kubernetes port. So the same thing as you mentioned like memory requests, CPU requests and limits. It also specifies the images which you are using in there. And the image basically defines what kind of software is in that one. And so, but it's level of specification is that the image level, I choose what kind of Docker can with the name or name inversion of a Docker container I want to use. Yeah, thanks. I mean, you see that part of the configuration. Do we still have it open? No, I don't, we can have a look into that. So Docker image here, this one Jenkins in both agent version four, three, four. And you could replace this one with something else. There's just, I think we'll come to it in the slides but I can also answer it now. So there are some default agents which for example include Maven or Node.js and include as well the small program which is needed to connect an agent to the Jenkins controller. And I normally prefer not to use this one but if I want to have custom tools in these agents and then I configure them differently as a Sidecar container. I think there's some example in the slides later. Thank you. So the first question people often ask how can I use my own Jenkins image? Because for production I would not recommend to download the plugins on startup because that creates some dependency to the outside world to the Jenkins update site. I mean, that one could fail and you also can't 100% be sure which versions of the tools you are installing. Therefore what I suggest is people just please create your own Jenkins image based on an official one and in that one, install the plugins which you need. Obviously you would need to install the plugins which GM chart depends on like the Kubernetes plugin and configuration as code but besides that you are free to install there whatever you want and then just use that plugin. How can it be done with the chart? Just configure it via a values file and set this one. I have a demo for that one. So here's my custom image. So you see that's just something I built. The last part is important these install plugins. You see I set this one to an empty array. This just tells it, okay, I already have the plugins in my image so please don't download any plugins because they are not needed anyway. So therefore I'm just putting this to an empty list and then Jenkins will not try to download something. And I'd love to use this one but in combination with something else. As I don't want to look up the random password all the time, I mean in a productive setup you would also configure like the default users authentication and so on using configuration as code. I mean typically use case is you have an active directory and LDAP server and OADC provider and you configure Jenkins so that authentication goes via that one or a simple GitHub authentication. And therefore you would not use these built-in admin user or password but for this case, I mean here I'm simply in the demos setting the master admin password to secret. So let's do an helm upgrade Jenkins Jenkins. I use the fixed password as one and as a second values I specify my custom image. Then connection to Jenkins will be dropped because it's a new image. So this one it needs to restart the Jenkins container. We'll take a short moment because Kubernetes just needs to download that image and afterwards we'll simply start up. And so you mentioned that I can bundle the binaries of the plugins inside the image. Now do I do that by defining a Docker container? Do I do that in a Docker file with a ref directory? What's my technique for placing the binaries of the plugins inside? Let's hope that looks like I made a correct guess. This is a repository for the Docker image from Jenkins and this one contains instructions how you use it and it also contains instructions how to installing more tools from Jenkins. So this one is a Docker file. You say user then installing additional tools and switch back to the Jenkins user and pre-installing plugins. Yeah, you've got also an example how you could do it. I mean three difference basically. So the newest one would be you do from Jenkins. Jenkins and you specify the version. LTS is like a floating text. So it always points to the latest LTS version. You could also specify a specific one and then just say Jenkins plug-in CLI plug-ins and afterwards you specify the plugins you have. There are two different way either like this one. Just say I'd like to have the Docker slaves plug-in and take the latest version please. Or like that one here specify the version number. So you know this one will be version 1.8 and the plug-in CLI or the install plug-in script basically will look up all transitive dependencies of these plugins and install them as well. And so I noticed that you've taken us right to the standard Docker image. So it supports the operations I beheld chart will use any Docker image that I define. That's great. Okay, so I can also then bundle my, I use exactly this technique and I bundle my binaries inside my Docker images thanks to this, very good, okay. Yeah, I mean, it's meant to be built with the official Docker image. You could use other ones, but then you have to ensure that for example Jenkins home Viabla set or so, I mean the chart needs to take some assumptions of where it finds things. Right, so the chart is optimized and tuned specifically to use the Jenkins, the official Jenkins repository. That's great, thank you. Yeah, so let's check if Jenkins is starting up here, looks good. So do a port forward once again and we should notice a little bit of a difference. So now I can use secret as password because I specified that one. One difference you see is that now here's some messages pop up. Well, so the background is a little bit different because I use, not sure. I think I used a custom theme for that one, but I'm not sure if I configured it here. Here you see some warnings because this image I use Bernhardt's Jenkins Kubernetes Secrets plugin and that one, it doesn't work at the moment because some things are not set up in the right way at the moment, but besides that, nothing changed. Now I have Jenkins with my custom image and that's the first use case I would have and something else to notice, you see here in my values files, I just specified values which differ from the defaults and I would say that's the best practice. What you see sometimes people do is copy over the default values. So I'm now just going to the hand chart and the chart Jenkins directory and here you've got the default values.  What some people do is just copy and paste this values file and just adjust the settings which they want to have adjusted. The drawback of that one is that it's hard to tell what are the settings which you want to customize and where do you want to stick to the defaults and therefore I rather recommend just put in your values for what you really want to configure and that's also way easier if you get a bug report and somebody points to me, this is a version of my hand chart. These are the values I used and that's my problem now. With that, let's dive into agent configuration. So we saw already in Jenkins where you can find this one. So Jenkins manage Jenkins and you follow up to the pod templates. The pod templates is basically the same as an agent template. There are different images involved in running agents. One is the GNLP. This is the one which is responsible for connecting to the Jenkins controller. If it's not defined, it will be injected automatically by Jenkins. So if you specify a pod and that one doesn't contain a container which is named GNLP, it will automatically inject a different container into that pod which is named GNLP with this default image. And then you can add whatever other containers as you want to that image. So as you learned earlier, a pod can consist of multiple images. And in these other containers, you basically can use whatever you want. The only thing you have to ensure is that the container is not killed. What do I mean by that? Quite often containers, if you start them, they expect a default argument or something and otherwise they just print out a help text or so. And in that case, you can have an exit code. So exit zero, exit one. And if in Kubernetes one of the containers exits, then the containers restarted. So you will see that your agent is never really connecting to the Jenkins controller because it's just restarting all the time. And very easy way is to just start the process within that container, which runs forever. What I see people typically doing is just starting cut, reading from standard input and this one will take forever. Makes sense? That does. So the GNLP container is the communicator. Other containers, do the other containers and the GNLP connector all share a single file system or do each of them think they have their own file system? No, they actually share the Jenkins workspace. Okay. By default. And another neat trick is that you can specify the default container in your pipeline. So for example, if you have multiple containers, I mean, if you don't specify something, then everything will be executed in the first container of the pot, which is a defaulted GNLP one. But let's say I have a GNLP container and a Maven container. If I have a Maven container, then probably it's a Java build and everything I want to do is to execute Maven. And I could wrap every command in something which says, do I have an example? I don't have it here. You need to wrap every command in a container and specify in which container this command can be run. I mean, that's also quite handy because you can combine multiple containers with different tools installed and just specify in which container the tool should be executed. And here with the default container, you can say, just use my Maven container or my tools container by default and stay away from the GNLP one. And using the Sidecar concept also provides some benefit in terms of isolation because this GNLP container uses a Java program to connect to the Jenkins controller. So that one is a JVM. If you will, let's say you also have a Maven build and Maven spawns another JVM. If Maven would run integration test, then that could be that that one spawns the third JVM and suddenly you've got like three different JVMs within your container and each JVM thinks, okay, all the memory is up to me, so let's just use it. And then suddenly you'll see an out of memory killed of Kubernetes because you specified on your build job that, for example, it should take a maximum of four gigabytes of RAM and the three different JVMs, they don't know from each other and everybody just thinks, hey, I'm alone here, so let's eat up all the memory. And if you use different containers for that one, then you could also set up these limits individually per container. And with that, I would say you get reproducible build because you can start, use the exact same container which you're using here without any dependency to Jenkins. Use the same memory and CPU limits locally, use the Docker run, share the local volume in that run your build command. And if it runs there, then it should also run file in Jenkins on Kubernetes. And that, so you're saying I can have control basically on a per container, how what its memory and CPU limits should be. So I could say, oh, I know Maven needs eight gigabytes, but the next thing that's going to do something with JavaScript in PM, I only need to give it three. And that is allowed. So we see some examples here. So you can configure them. One example is with, why are they, this one is a values file for the hemp shot. So you can use agent pod templates. And let's say I want to configure a Python agent. Here you see these arcs, which I meant in cut just to ensure that the container stays running. And here you see CPU request for CPU and memory and also the limit for CPU and memory. And let's try this one. So I have an agent pod template. So I'm not cheating here. That's exactly the same content. So what I do is send him upgrade. I'm using the same as before. So the same image and this fixed password and also these, we had to move this stuff. Agent pod template, YAML. So let's execute it. So see everything is running. My Jenkins should still be running. So no restart, no nothing. If I go to manage Jenkins, just ignore all these errors for now, you will come to that and go to configure clouds and my pod templates. And now you see, besides the default, I also have the Python pod template. With exactly the settings which I configured. So with Python 3, this one is named Python 3, everything's fine. And here are the requests CPU and memory limits, which I specified in the conflict file. And the way you passed those values in was just by adding one more values file to the command line. Just like this. Very nice, okay. And nobody needs to fiddle around in the UI because I mean, that's also error prone. I mean, if some person makes a change, you forgot something or makes a manual change and nobody else was aware of and now something is not working. And these values file I can simply put in the version control. And if I want to use this one, I'm just modifying my demo now. So label Python, save. If I didn't make any mistakes, then it should start up a Python agent. Okay, and so, and will the agent be inside that pod and will it show up as a separate? As a separate? It will show up as a separate pod. So this one is still the Jenkins one. And we will see a separate pod and that pod should have, I'm not mistaken, two containers in it because what we specified. It would have one for the JLP container and one for the Python container. Exactly. And then in terms of diagnosing of logging, reading logs, logs are available, you showed us from Jenkins, are there any other places where we should know to read logs or to look for logging? I think that's the most important part. I mean, you could just use the logs from the covenators. Oops. So logs, my release Jenkins and dash C Jenkins. So that's a Jenkins controller itself and you see controllers in there. And you see, oops, cannot resource secrets or something else. So it doesn't want me at the moment.  We continue. So the, perhaps the next one, what we can do is someone introduced in the chart another maintainer, basically an additional agent settings. The difference to the pod templates is that this one inherits settings from the default pod template. So everything which you don't specify here, but which is already specified in the default template will be taken from this one. And that's also quite easy. So let's try an example and create a Maven pod. Let's see agent pod template, I think. No, this was the wrong one, agent inheritance. So here we go. So this one you see here, Jenkins, GNIP, agent Maven, Helm upgrade. And I'm just providing another values file with my agent inheritance. That's it. And if I do a refresh in Jenkins, I mean, the configuration is already there. Configure clouds. Here's my pod templates. There you see my Maven template. That's it. That's elegant, powerful. Okay. Let's give it a try if it works with the Maven thing. Because then I can show something. While it's trying to starting, I can explain it. There are two settings which are interesting.  The pod retention basically tells you if you want to keep the pod or not. Always means just always keep it, which means these pods will be kept in Kubernetes around no matter if it was a success or a failure. So you wouldn't need to keep insured by yourself that you clean up these pods. Otherwise you will end up with a huge number of pods in Kubernetes, which are basically just finished. Then you can say, never, this is a default. So once the build job is run, just delete the build pod and it's gone. Or you could say on failure, that's interesting if something fails and you'd like to be able to check the Kubernetes logs afterwards. And you could see on failure, the pod will be kept there and that's there and everything's okay. And the default setting is just you don't specify the setting on the agent, but you specify it on the cloud. So you set a default for Kubernetes and with this one on the agent, you could basically override that setting for a specific agent. And does on failure in this case mean any non-successful run of the Jenkins job or is there some higher level concept that's related to just a failure in the pod that's not necessarily a failure in the Jenkins job? I think it's a failure in the pod. So if the pod doesn't start up correctly or provides another error and doesn't connect to Jenkins or whatever happens. So it's not really related to the build job itself, I would say. Great, okay. So I don't have to worry that, oh, somebody has failing tests, the job is unstable and I've got a bunch of clutter from pods that are being retained by the on failure setting. Yep. And the other thing here is these idle minutes. So you can say everything was successful. Don't delete it directly. At least keep it idle for a specific number of minutes. And that one provides quite some benefit. This one worked by the way. So as the job was executed, you could be sure that a pod was created and what we could use is these idle settings and agent inheritance dash key pod idle. That's my YAML file for it. And build now. You see the first time it takes quite a while to start up this pod and until it's connected to Jenkins and everything works, we could also check the pod templates. I need to refresh here because Jenkins doesn't live reload but the setting is still there. So we will see these idle settings is in time and minutes to retain agent when idle. This one was now set to 10. And you also see the pod retention which was set to on failure. Exactly what I specified in my YAML file. And now, so time to, if an agent is retained when idle, can it be reused by a later job? Is there a risk and a benefit of reuse or it's not actually reuse, it's just it's idle and then it's deleted? No, it's reused. So you see here the Maven pod, the job is finished. This one is still running, it's 42 seconds old. So let's start another job. And you see how quick that was. Okay, so it used it as though that agent were a cache for it, that it could reuse and therefore all the... It's basically inside the agent is preserved. It's an agent which is connected to a Kubernetes. It's already connected. So the build job can start immediately on that agent if it's free. And as in this case, I'm just executing a hello world on it. I can do a build now and you will see it already succeeds because there's no time to start up an agent also. Benefit is that you don't have to wait for the startup of the agent and you could also benefit from caching. So let's just assume you have a Maven build and you download all the dependencies. These will be cached on that agent. And that can be a benefit or drawback. However, you'd like to see that. Right, you're getting all the benefits of caching and all the dangers of caching that go with if you expected your cache to be flushed. Thank you, that's very good. So Xena, my apologies if my questions are distracting you or slowing things down unacceptably. You are welcome to tell me to stop asking questions. Don't be shy. No, dear, no, please go ahead. But it's good to hear that you're still with us. I am actually, I'm taking notes. And Xena, I assume you're okay if we continue this. We're at an hour, but we had talked about going another 30 minutes if needed. Yes, it's also fine. I mean, whatever works for you when you say you need to leave, we can stop and schedule for another meeting. I actually have a question for you. I want you to finish before I ask my questions. Okay, how to use agents in the build? Here finally it comes. So this is a label, should be lower case. It was the auto correction, Maven and Echo Hello World. How to use different agents without pre-defining them. So what you've seen so far, I just pre-defined those agents in the Kubernetes configuration. And it has quite some benefits in terms of you have standard agents configured on your Jenkins, but also quite some drawbacks. I mean, if now I'm changing any of these standard agents to say from Maven 3.5 to Maven 3.6, then everyone who's using that finally has Maven 3.6 and all the issues or whatever is introduced by that one. And there's also a different way that's also quite handy if you want to test something out. So you could either specify agents inline, what you see here is just a jammel file and this is a Kubernetes pot spec which you can put in this jammel. The same command cut service account. That's a totally valid agent configuration. And you can also do it by reference. If, let's say your build pipeline is stored on the version control, then you could also specify like here the build pot jammel in the version control just say the jammel file of where the build pot is in. The benefit I see is that if you have this one separately then you can use of course all these syntax checkers for Kubernetes manifest and it doesn't make your job too long. Now, Torsten, you became a little quieter for me as you were describing there. I don't know if your microphone moved or something changed. Is it better now? Much, yeah. Okay, sorry. So understanding agent permissions. Understanding this one in Kubernetes is quite important. So which permissions do agents have in the cluster? Nothing special. If you understand Kubernetes, well it's just the permission of the service account of the agent pot. So I mean every pot you start in Kubernetes is using a service account or whatever permission that service account has in Kubernetes, that's what the pot can do. Which service account do they use? So if nothing is specified as service account then default service account of the namespace is used. How can you specify a service account? Why are the agent template specification or why are the pots back? So I think we saw it in the examples here. I thought that in one of them I have a service account specified if not be with me yet. It's just one value saying service account dash and the name of the service account. And how are permissions granted to a service account? There are two different ways. One is called as role binding. The other one is cluster role binding. So in Kubernetes typically you have roles and these roles define what you can do. And there are two different types of roles either roles or cluster roles. A role is limited to a namespace and the cluster role as a name implies is valid for the whole cluster. And the role binding is basically can also work just for a namespace or for the whole cluster. That's what I wanted to show here. So and why it's a bad idea to run agents in the Jenkins namespace? Remember that the Jenkins Kubernetes plugin needs Kubernetes permissions as well because it needs to start agents and it also needs to stop agents later on. So these are the permissions which are used. So this is a role which is also created by the Helm chart. And here you see that for example, that on pods and pods exec permissions can be created, can be deleted, can be patched, updated and you can also exec into a pod. And if this one is the commission which the Kubernetes plug in needs and I would now run agents in the same namespace as my Jenkins controller, then I could simply specify, please use the service account to which this role is bound. And with that one, I could execute into my Jenkins pod. So into the Jenkins controller, I could delete it and could do all kinds of crazy stuff within Kubernetes. So therefore I would say, or I personally prefer for production environments to have my agents running in a separate namespace than the Jenkins controller. And also be a little bit strict on what you allow agents to do in the Kubernetes cluster. I mean, you saw in many older examples and getting start guides for Kubernetes just giving something cluster admin permissions. I mean, that always works because cluster admin can do everything. Yeah, but you probably would not recommend that for production. And I also see some other benefits of using a dedicated agent namespace. One is what I already said is a better security. And the other one is that you are able to limit the resource consumption on the agent namespace. I mean, Kubernetes has a concept of specifying limits per namespace. So that you could say, whatever you specify in the pods, the sum of everything, you're not allowed to use more than this amount of memory in this namespace. And like this one, if you have the separation between the agents and the controller, you can specify different limits for the namespaces. And it's also like that the agents don't directly fight for the same memory as the Jenkins controller is using. How does it look like in the deployment? I mean, for this chart, just you have to specify one value, slave Kubernetes namespace. I mean, you notice there are some deprecated terms and we should also get rid of them. So renaming it to controller and just agent. But in the end, what it does creates these rule binding for these schedule agents in a different namespace. And with that one, all the agents, which will be created are then just spawned in that separate namespace and no longer in the namespace where Jenkins is running. Okay. So with permission to the Hamchank grant, yeah, only those which are needed to run Jenkins and to spawn agents and to reload its configuration, everything and optionally to read secrets, which permissions do I need? I mean, that really depends on your use case and what you want to do with your agents. I mean, if you just want to do an Maven build and don't want to execute anything beside that in Kubernetes, you're all set, everything is fine. If you also want to do deployments to Kubernetes, then you would need to set up a service account and grant that service account permissions, ideally just on the namespace on where you want to deploy something and not on the whole cluster. But it really depends on your use case on your setup. You just have to think about and that's why also the Hamchat doesn't provide any more defaults there and if set one up to the user, make sense? Yes, thank you. Credentials, I wanted to mention some words on how to manage on credentials on Kubernetes. There is these awesome Kubernetes credentials provider plug-in and it allows you to use just normal Kubernetes secrets. So this one is a Kubernetes secrets. All it has is some annotations and some labels. So this one tells it, this one is a description for my credential and it's a credential type username password and it also has examples for other passwords, just only tokens and so on. And this is all it needs to create secrets. The thing is that that credential that plug-in basically needs to read these Kubernetes secrets so that can directly show them in Jenkins. You saw earlier that I use the custom image here and we got many changes regarding the credentials plug-in for Kubernetes secrets. Now you know the answer, it's complaining because it's not allowed to read secrets. If you go to manage Jenkins and manage plug-ins, you will see that it's installed. Oops, that Kubernetes credentials provider plug-in here it is. And if you go to manage Jenkins like now and go to manage credentials, you see also here there are two different credentials provider. The first one is the built-in Jenkins credentials provider which just encrypts credentials and stores them on disk. And the other one is the Kubernetes credentials provider. And I'd love to show you a short demo of that one. So what we need is RBAC read secrets. That's another values file and this one will set up the necessary role and the role binding so that Jenkins is allowed to read secrets in its own namespace. So let's hope that credentials YAML is the correct file. Yeah, it is. This one I made now some changes to my role bindings and roles. So you now see I have three different roles. One is for the cast reloading, so reloading configuration. The other one is about reading secrets. This is what I need for that plug-in. And the other one is to schedule agents and I could also get the role bindings. Oops, and here you see that these are bound to my release and basically to the default service account of that one. What I need is just to restart Jenkins. I'm doing it the hard way. Please don't do it in production like this. I mean, not get but delete. To answer your question from earlier, so here's my persistent volume. This one by default just has eight gigabytes. And if Jenkins starts from scratch, it's going to reuse that system volume. You just killed your Jenkins leaving its file system in place and you'll now start a new Jenkins and it will reattach to that file system. Exactly, so that's basically the same as you would restart Jenkins on a virtual machine. So the Jenkins process itself is gone but the persistent disk is still there. And if you restart Jenkins and your file system is not corrupted then it will start up successfully and everything else is there.  You also see like the existing executions. So all that was thought on disk and it's working. Now you see here a little bit less warnings. Oh, there's a newer version of Jenkins so I should update my demo. And we now can go to manage Jenkins once again. And manage credentials. See nothing in here. So this one is a secret which I just copied from the examples of the Kubernetes credentials provider. I do a kubectl apply F, F says file. So I'm creating that secret. I can check that the secret is there. So another test username pass that was the name of my secret. Go to Jenkins, refresh. Here you see the ID isn't just the name of that secret and that's my username pass. That's it. And if I'm going to edit that secret and let's say I want to have a different description. Hello demo, refresh. There you see that's my updated secret. And the same happens with the content but the content is not so nice for a demo. That's how you can manage credentials. Now the important part, configuration as code. You already saw it in practice. It's using the Jenkins configuration as code plugin. And with this one here you have values. So what we could do is just provide a welcome message. This one here is basically the configuration as code. So in Jenkins there's a property called system message and I can say welcome to our CI CD server. Remember Jenkins has no message, no nothing. You see I'm just adding values file so you can use multiple. Of course you can combine all this into one but then it's not so easy to show it in the demo. So welcome message. Here we go. Refresh of Jenkins, welcome to our CI CD server. That's configuration as code. How does it work under the hood? As I said, there's the Jenkins controller but for this one it's important is the Sidecar controller. This one watches for the config map with these. If there are new ones or edited ones or deleted ones whenever they are, it copies the configuration into the Jenkins home folder into a directory called I think cast configs or so. We would need to look it up and it just triggers a reload in Jenkins. And this triggering of the reload is nothing else than if you would go to manage Jenkins configuration as code reload existing configuration which just checks on the persistent disk what configuration is there and reloads the stuff. And with the config maps CM is a abbreviation for config maps. So there was one welcome message and you saw here on the slides what I use in the values file. This key welcome message here is used for the config map. So if I edit this config map manually now outside of the hand chart, don't do it. That's just a demo. You see that's an API version, API version VR. It's a config map. This is my welcome message. And I say, there you see already demo. This is just how it works. How does it work internally? I mean the Jenkins controller JVM is started with Java ops, cast reload token. This one just uses the name of the pod so it's kind of random. And the sidecar reload does nothing else than a post request on localhost. Localhost is possible for it because these two containers are in the same pod. So therefore from this one, I can call localhost and I'm then referring to basically Jenkins. Say reload configuration as code and I provide these cast reload token which is defined there and this is how the reload works. And now the beauty part let's remove the welcome message. So you saw earlier, what I can do is QCuddle exit into my pod and execute Agile J commands in there. And here I'm just listing my VAR Jenkins homecast configs directory. The name of my pod needs to be updated. You see there are two files, just Jcast default config. That's where all the default configuration of the Helm chart is in. And the other one is just the welcome message. So now let's do Helm upgrade again and now I'm removing my welcome message from the values file, typing enter. What would you expect now to happen? Is the welcome message still there or is it gone? I don't know, okay. It's still there. Everyone can ask why? If we check on the disk, basically only the Jcast default config is there and the other one, so the welcome message YAML is really removed. So the configuration file is gone but the configuration is still there. Why is that the case? Because the configuration as code plug-in only configs things which are specified in the YAML. Whatever is not configured in the YAML will not be configured in Jenkins. If we want to get rid of the welcome message, we would need to set a new welcome message. Yeah, we could say, just put in here a system message with an empty one. And if I do this one, remove welcome message, do reload, it's gone. Because now the configuration as code plug-in picks up that configuration, configs Jenkins accordingly and the configuration is gone. And now I could remove this file because if it's still there or not there, it doesn't make a difference to the configuration. So it stays the same. I mean, simply it doesn't change anything. I would say that's a typical pitfall people don't think about. I explain this also in the slides here. There's another thing which you could do. So sometimes you want to configure something and maybe even template it outside the hand chart or from somewhere else. You could even use your own configuration files. I mean, the Sidecar container is just looking for config maps which have a specific label on it. And the label here at the first part is the name of the release which you provided. And the other one is just Jenkins-Config and the value of true. So if I provide another config map completely outside of the hand chart and auto-reload will pick that up and configure Jenkins accordingly, makes sense. So for example, here I've got just a custom config map. This is not in the hand chart. So I don't do a helm this time. So kubectl apply my config map and configuration is there because this config map just has this annotation. So my release Jenkins-Config true. And the important question is how do I know what can be configured via Jenkins configuration as code? I mean, not everything is documented in the hand chart. I mean, the hand chart documentation could need some improvements anyhow, but it also doesn't make sense to repeat configurations options from somewhere else. I mean, it's the same with the Docker image. I think it's best if it's documented that the Docker image repository, we should only link to that one. And it's the same with the configuration as code. So the best way to find information on that one is just go to its GitHub repository. It contains two interesting directories from my point of view. One is the docs sections. Sorry, not docs, the demo sections because here you've got quite some examples to configure different plugins. GRO, jobs, key clock, whatever, or LDAP for example. And then you see how you can configure an LDAP server. Just be careful with these kinds of security configuration because it's quite easy to kick yourself out of. But the beauty of configuration as code is you could also revert it if you'd like to. So demos directory is one source of truth. And the other one, which is interesting is the integrations and the integration tests because that one also contains YAML files which are used in their integration tests. And of course, if it's used in the integration test, you can be pretty sure that it's also working if you use this file for configuration. And the examples and integrations contain sometimes different things. So therefore, I think it's useful to look into both. And another source of truth is just a bit in documentation or off that plugin. So if you go to manage Jenkins configuration as code, you can do two things. One is to view your existing configuration. So this one helps if you just configure something manually, view the configuration here, check how it looks like and then do copy and paste and to manage it via code. Or you could also download this one and here's documentation as well. And this one depends on which plugins you have installed. There are different options and then you can simply browse through the documentation here which keys exist and which values are possible there. That's about configuration as code. And then I could talk about some chart improvements which are planned. I mean, one is the deployment to replace it with a stateful set. Then I'd love to get rid of all the XML configuration options. I mean, the past configuration as code didn't exist and therefore what the help chart is is template XML configuration files. And that one is of course on the one side quite buggy and on the other side, you're not able to live reload things and you're also not able to just overwrite the XML configuration files because it might contain more configuration options than what you'd like to change. So you configure something on Jenkins startup and everything works fine. Then the user starts to configure something in the UI and then it would be a pity if you just overwrite their configuration and everybody would be annoyed. And if you're not doing it, then people are annoyed because you set something in the YAML file and you expect that the configuration is there but actually it's not because we are not overwriting XML configuration files and therefore I'd like to get rid of that one completely and just configure things by a configuration as code. I think that also helps to promote the configuration as code which is really nice to even further. I'd love to get rid of all these offensive terms replacing master, slave and so on to use web sockets to connect agents at least as an option which you can enable and maybe in the future make it a default so that it also allows agents outside of your cluster to connect to your Jenkins controller and it also would make the architecture of the Helm chart a little bit simpler because you could basically get rid of the service for these port 50,000. I'd love to get rid of the Helm as a chart label so there's this label is on many resources which contains just the name of the chart and the version of the chart. And I mean, some people are using GitOps when deploying their things so basically they are using Helm not to install stuff but just to template Kubernetes manifest. And with this one, they have basically whenever there's a new chart version, this file is changed even so all the content stays the same. And another change which I'd love to get in this to use Jenkins plugins CLI to download plugins. That concludes my talk. Now it's up to you to ask more questions. Thank you so much. I think this was, I don't see a lot of content that we can include into the presentation for the Jenkins and Kubernetes. So, who can hear me? Yes, I just put it a little bit louder. Okay, so my first question please. Can you show me where you installed Jenkins using Helm? Is it there? This is where you installed Jenkins. This one here, do you mean Helm installed? Yeah. Helm installed. So while I was going through a lot of resources on installing Jenkins, I saw them in the tool here. We checked the value file, Jenkins value file while installing, but there's nothing like this here. So how do you include the values or apply your values while installing? Like this one, something like this. Yes. So we could even put in an alternative here. So you saw me, I used all the time Helm upgrade install. Yes. So you could also use this one. The upgrade works on upgrading name and existing chart. And if I provide dash dash install, then you can also use it to do the initial installation. So one could even use the button below in favor of the first one. Okay, wait, so this Helm installed, installs Jenkins using the official Helm chart, right? Yeah. So at this point, if there's no creating any persistent for new year, creating anything specific here, just using the official values of this command. Exactly. So if you don't specify anything in here, it's just using the values which are specified in the chart. So everything, a chart is nothing special. So all you see in the, see here the Helm charts directory. I mean, in every Helm chart has in this chart, Jamal, just the version. And this is the version of the chart. And then you have a number of templates for templates like for these, for the deployment itself and so on, which can be templated based on these go templating language. So you see here some if, else statements and so, and as values, it's using here these values, Jamal. And a release chart is basically nothing more than create a target set out of this directory and put it on some directory. So you could even download it manually and just untie it. So if you saw here, this chart is now version two H two. If I would download that chart and extract it, it would contain this values file. So if you don't specify any values, it will take the values from here of the chart version you specified. I'm sorry for this. I'm really sorry for the queue. In this case here, I'm just use a kind. Kind is a Kubernetes in Docker. You can also use a mini cube. I think both will work. I think you had an issue or mentioned something there about the volume which is created. So basically did the home persistent volume of the chart you see in here. What you used is a feature. I mean, you could either disable the persistence. That's one flag here which you can set. Then we won't create a persistent volume claim or you could set an existing claim. So basically refer to a volume which you set up in upfront, but you see other values. So you can say the persistence style or the storage class name. What you use was a host path volume. I mean, that's, I would never use something like that outside of the demo and not even sure if it, it's a good point to mention it to users because they might try to use it and stick to it. And actually on many production Kubernetes cluster I would expect that host path volumes are discouraged or for users even disabled because that would allow you to mount the host volume of the nodes where all the Kubernetes ports are running. Other questions? This one here. Or these port templates. How to do it, how to do it using help, how to command using help, how to spin off agents. Ah, this one. So basically, what I used was in the end just this so I can share with you later on in the other channel. So it's just an help upgrade and to say dash dash values and specifying the additional values file. And all I did was did a copy and paste of this one and stored it in a file and specified it via values file. Okay, thank you so much, interesting. Well, can you share this slide so I can see if you think you can have any other questions? Absolutely, I think it's already shared in the Cuban Jenkins CI Docs channel. I can reshare it there once again and also send you the link. And if you have any questions regarding the Helm chart or outside, I mean, just let me know, ping me in that channel, ping me by a direct message and I'll try to help you. Cool, then that's it from my side. Torsten, thank you so much for hosting this session. Thank you, thank you. I'm confident ZNAP gain benefit but I know how much benefit I gained. Thank you very much. Yeah, you're more than welcome. I mean, as I said, I would also be happy to do something like this or similar in the Jenkins Meetup or so if desired. I mean, totally up to you. We definitely want that. And so that's great that you're back. I will set up with you separately. We've had a Jenkins on Kubernetes theme in our Meetups recently and so we'd like to continue that and have you do this presentation there.  Because I would love to do it not next week because we've already got a Meetup schedule next week, potentially the following week or the week thereafter. Yeah, I don't really care about the timeframe when it is. I mean, if it's later than probably we have a new release of the help chart. Well, I could update something. If it happens before that release then I could motivate some people to join. I mean, whatever works, we'll find a way. Excellent, thank you, thank you. Zinab, any other topics that you'd like to cover before we end? Okay, then the recording of this session will be posted to YouTube and available. All right, thanks everybody. Thank you, thank you.


I'll tell you, the more I just stayed awake last night thinking about this thing, the more I think of it, I don't know what in the hell it would look like me getting into another career. I don't see what we can ever hope to get out of there with once we're committed. I believe the Chinese coming into it, I don't think that we can fight them 10,000 miles away from home and never get anywhere in that area, I don't think it's worth fighting that far and I don't think we can get out and it's just the biggest damn mess there is. It's an awful mess. One thing that has carried me, I'm worth to me, what is Laos worth to me, what is it worth to this country? Now we've got a treaty but still, we've got a treaty but still everybody else got a treaty out there and I've done the thing about it. Now of course if you start running the Communist, they may just chase you right into your own kitchen. That's the trouble. And that is what the rest of the, that half of the world is going to think if this thing comes apart on us. That's the dilemma. That's exactly the dilemma. But everybody I talk to, it's got any sense in there. They just say, oh my God, please give us those. Of course I've read the Nashville stuff this morning and it's just milk toast as it can be. He's got no spine at all. But this is a terrible thing we're getting ready to do. President, I just think that because it's the only big decision in one sense that this one is one we're having, that we either reach up and get it or we let it go by and I'm not telling you today what I do in your position. I just think the most we have to do is to pray with it for another while.