Hello everyone, welcome to the Jenkins Infrastructure Team Meeting. Today we are on the 12th of September 2023 around the table. Today we have myself, Demi and DuPortal, Mark, wait, might or might not be able to join us. Stéphane Merl is there, Bruno Vartan and Ashutash Saxena, is that pronounced properly? Yeah. Cool, so welcome. Let's get started with the usual announcements. First of all, the Jenkins Weekly 2.4, no it's 3.1, is released. So the packages, Docker image are out. So last release elements as usual will be done by either the documentation SIG, mostly Kevin and Mark, a bit later, will be done later today. That means the Jenkins Infra Team is ready to deliver that new version to Infra-CI and Weekly-CI. Mainly, Stéphane Merl, you are ready to go. Usually it's Wednesday morning on the European time zone. But yeah, you have the go, you can proceed. CI, Weekly-CI. That's all for the weekly release. That's the first release without us mirroring the Apache Central Maven repository. So that means that's really going in the right direction. Since we removed them, we didn't have any major thing. We only released plugins. So next step will be the LTS release that should go fine and security plugin release. Do you have other announcements? Nope. Okay. Yeah, maybe. Oh, that would be the major event. Sorry. No, I was thinking of the world. DevOps world? DevOps world. Yep, anonymous calendar. So upcoming calendar, we should expect the version 2.432 of Jenkins next week. It will be in September 19. That will be the same day as your team meeting. Don't remember for the next LTS. So let me check last week's notes. Yes, might cheat it. 26 for 26, maybe. No, not 23. Oh, the next LTS release. Yeah, I'm wrong about the number. It's so the weekly release today was 2.423. So you are correct. Next week will be 24. And the next LTS release of Jenkins will happen the 20 September 2023. You can remove RC. Yes. Absolutely. RC was done last week. And that version will be 2.440.2. Do we have announced Jenkins advisories? Unless you have something about the weekly or LTS release. Nope. Okay. So we had Jenkins plugin security advisory last week, last Wednesday, during our past milestone, which went very well. There wasn't any plugin that we are using, except job configuration history on one of our private controller. But that's not a concern and has been updated. So nothing to say about the release, the security releases. Last elements, next major events. DevOps world. DevOps world happened in New York this week, this third day, and it's Friday. Then Chicago 27 September, and it will be in Santa Clara in October. I don't know for the Asian and European steps, if they happen or not anyway. So we'll wait for more to give us more news next week. I think London is already planned. And I think it's January or February next year. Cool. Thanks. So if you want to meet some community members, that's one of the opportunity. And there will be a session. Sorry to interrupt. There will be a session after each day of conference, which is called let's talk about Jenkins. So it's not ask me anything. It's just just talk about Jenkins and one community member will be there to discuss Jenkins with all the participants. Let's talk about Jenkins. Nice. Thanks. I believe the first time has been confirmed to happen. Yes. Three and four of February, I think. February 2024. Okay, that's all for the upcoming major events. So before we jump on the task, the infrastructure task, it's for you Ashutosh. It's your turn. So can you explain us your requirements under this recording? I will take notes. If you need me, I can lead you and let you share your screen or you can guide me on that screen share. You drive. Yes. I think you have been frozen. We can't hear you. Is it only me? I can't hear him. Ashutosh, I think you have been kicked. Yeah. You don't seem to be muted, but we can hear you. Okay. Better. You're back. Right now we are using GitHub Actions for uploading Docker images and my personal Docker of account for hosting the images, which should not be trusted by users. So we need official Jenkins account and Docker Hub for images and for testing the images and if they're working or not, we'll need to shift from GitHub Actions to Jenkins CI. Jenkins.CI. So today, you are already using GitHub Registry to store the Docker images. Is my understanding correct? Yes, but they are hosted on my personal repository right now. Oh, I was speaking about GHCR.io, which is a container image registry like the Docker Hub that can be used publicly or privately like Docker Hub, but it's inside GitHub, which has the advantage to, if you use GitHub Action, you immediately can start working on building the image on GitHub Action and then immediately store it and distribute it through that registry as a first step, of course. Right now, I don't think we're using that. We are just loading the images directly to my own. Okay. Okay. I believe it will be safer for you and for the project to start with GHCR.io. The reason is that anyone who has the maintainer permission on your GitHub repository can also deploy an image based on the build from GitHub Action. So not only it will be safer as a first step, again, it's a first step. Not saying we don't have to move to check in something, but right now it's a first step for you because that will allow you to use a Docker image that can be shared with others, not only you. It's technically possible with your current setup, but that will be easier for handling the project to avoid staying on a one-person project. And also that should unblock you until we make a decision on where to store the image for that project and do we need to store it as an official Jenkins asset? Does it make sense for everyone? Yes. So now about the official Docker Hub organization that are storing images for the Jenkins project. We have two main organizations today. We have Jenkins. For instance, you have the official Jenkins controller image, which is, for instance, that's TAG. That's the Jenkins organization and Jenkins infra. But Jenkins CI infra, sorry. CI infra, yes. So that organization has some images such as, let's say, this one, for instance. These are two examples. Each organization are mapped to a corresponding GitHub organization. Maps one to GitHub organization. We have Jenkins CI for the first and Jenkins dash infra. I will add the link on the notes for richer notes later. So what is the difference between those two images and how do we decide when the artifacts that we produce are consumed directly by Jenkins users, such as the official Jenkins image, official for agents controller, a plug-in HPI file. Then we decide to host them on the Jenkins Docker Hub organization with the code on the Jenkins CI GitHub organization. When the artifact itself is not directly consumed by the Jenkins users, then it means it's mostly used on the Jenkins infra code organization on GitHub. And if it produces a Docker image, it will be stored on the Jenkins CI infra corresponding Docker Hub organization. The example to make it easier to understand is the website Jenkins.io. The source of Jenkins.io is on Jenkins infra slash Jenkins.io, and it produces a Docker image, Jenkins CI infra slash Jenkins.io. That image doesn't exist on search for it. We have something else, but that's the idea. That website is consumed by Jenkins and user, but that the service and itself are not the artifact. We don't have the need to share or to provide to the community the official image. We are an open project, so the source and the image are available to everyone if they want to contribute, if the community needs to take it over. However, in terms of operating the service, that's the reason why the Jenkins official website is not under Jenkins organization, but under Jenkins infra organization. Is that general explanation okay for everyone or do you have concerns or question about that explanation? Okay. Good explanation. Thanks. So that means now we have to decide what will be the destination of the images from the GSOC project for Ashutosh. This will determine where do we store the source code and where do we build and release based on the trust level. The question is, will the Docker images produced for that project be consumed directly by Jenkins users or will be indirectly consumed? To me, it's directly consumed by newcomers, so new Jenkins users. Yes, this mostly would be directly consumed. Most of them. Okay. So that means that anyone going on Jenkins I.O. will have a tutorial and so they will follow the tutorial and we expect images in the name Jenkins slash something, column something to be present there. Is my understanding correct and is that assertion looks good for you? Yes. Okay. So that means we want, so that's the first step for you. We will need an exhaustive list of the images you want. So the tags, we don't care about the tags. I mean whether we have a 1.0 or latest or whatever, it's a trap. We don't want the latest tag. We should not produce any latest tag. We should only produce version artifacts. But we need an exhaustive list. So I guess we will have something like Jenkins, I don't know, tutorial, controller, whatever. That's the kind of information that we need an exhaustive list because we will need to then check with the person, the way we are administrating that part to create the repositories, allow permission and start the process for deploying these artifacts. Also, we will need a list of, for each of these images, what will be the expected name of the repository? Is it a mono repository producing a collection of images? Is it one repository per docker image? Something else. Is my question clear, Shutosh? Did you hear it correctly? Yes. I got it. So it looks like you have sound issue. I didn't hear you answer to my question. I got it. Can you hear me? Yes. Can we hear you? So what is the plan for the source code? Do you want to have one GitHub repository that will deploy every images? Do we have one repository per image? Right now, I think the tutorials part should be a separate repository that contains images for tutorials and another one for the main docker installation for Jenkins. Bruno, what do you think we should do? You need to justify the technical reason why we need multiple repository over one repository, though. I think what we have for the time being on your docker hub repo is one repo with different images. And I think we could use the same thing. Just one repo should do. And we have various images and sometimes various tags. But I think mono repo would be enough. Everything is technically possible, but must be specified prior to creating the assets. Mainly, the next question, once we know the source and the destination, then the third element is that you have to specify. So this free question needs to be written. We cannot just have the question here. So here it's more brainstorming with me giving tips. But here you need to specify the lifecycle of releases, which means you need to say when do we need to trigger a new build and deploy a new version of one image? Whether it's a mono repo or multiple repository, what are the conditions leading to a new version with a new tag? And then once that image is deployed to the docker hub, what would be the conditions triggering the whole let'supdatejenkins.io website? Does it make sense, Ashutosh, for you? Or do you want more? Yeah, the thing is for the time being, we are overwriting the image whenever a change is made. So I know that's not good practice at all. You know, it's like using latest, but even worse. We have Dependabot working. We may install Update CLI one of these days. And I think we may have release drafter, but I don't know if we should rely on... You are defining the how. I'm asking at your level question, folks. What would be the condition to trigger a new release? Is it a new dependency that we track is available? Or is it once a week? Or is it when a human decides to make a release? Or is it something else? It's more that kind of question because the rest is only technical details that I have no doubt that Bruno Ashutosh and other will solve. Yeah, so whenever a human being decides it's time to make a new version because we made some modifications to the Dockerfiles, for example, but also when a dependency is updated, like, for example, a new LTS, then we have to update. Okay. Depend... New dependency is available. Okay. And then same question, but we don't need an answer now because it doesn't... It's a separated process. So it's... I don't want to block Ashutosh from that part, but you have to think about that. Ashutosh, you are producing images. What is the value of these images? It's only if someone consumed them. So you have the GSOC part and I am a voice of the Jenkins project that say, hey, that's cool. You already have the value as the GSOC part, I hope, but that's a discussion that I don't have any voice on. However, handing over your project to the Jenkins community, that could be you working for the community, that could be someone else. It's not about who and not about how, but more about what again. And in that case, what I'm thinking about is what will be the conditions and you have to think about that to be sure that once you have a new image or the change, what will be the process to deliver that to the end users because pushing an image to the Docker hub is not enough. Because right now for the GSOC, you chose to override using the latest image pattern, but that won't be the case for Jenkins save. And we must not. Does it make sense? Yes, it makes sense. Cool. So it's a new constraint. It's not something that you had to resolve until now, so that the context changed. So it's not that you did bad or did wrong. That's absolutely not. It's just a context change to think about a different context, to make the handover from a GSOC project to something widely used. I let Bruno guide you on the steps, what would be required and what is not a requirement, so a nice two-up thing, because I don't have the scope of the GSOC fully on my mind, so I delegate to Bruno and Jean-Marc on that part. But if we want that work to be pushed on the Jenkins project, these are requirements. Okay. I think we have all elements to answer you.  And then we will have to ask the Jenkins docker image administrator to create the repositories and have the proper token if needed to allow building these images, which leads us to the infrastructure parts now. So as an administrator of the Jenkins infrastructure parts, but I don't have administrative permission on Jenkins, I can only work on the docker part for you. So my proposal is that now we are going to open a Jenkins-infra-ldesk issue. So Jenkins-infra-ldesk is a GitHub repository owned by the Jenkins infrastructure team used for everything that has a foot on the infrastructure or docker hub or GitHub permissions. So we have a centralized location where to track these kind of requests. So that means, Ashutosh, we'll have to summarize with the help of Bruno and Jean-Marc everything we say today and you will have to open an help desk issue that will be the action of it. Because then from there, we, the Jenkins infrastructure team, will be able to contact and ping the different actors and have a centralized follow. Is that clear? Yes, it's clear. You will have a CI that can be available on ci.jenkins.io, the public Jenkins. This one will be easy, a Jenkins file. We action from Jenkins-infra-team to track the job for builds. So we will create the job somewhere on ci.jenkins.io. We'll give you the link on the help desk issue. You add the Jenkins file on a pull request and automatically ci.jenkins.io controller will track and give you checks on each pull request and each commit on the principle branch. But that's only ci. The cd part, the deployment, will be on a private part that you won't see. I'm still not sure which controller needs to take care of ci. We have two controllers that are able to push to the Jenkins Docker Hub. We have trusted ci.jenkins.io, which role is to manage the update center and the official Jenkins images. And we have infrared ci.jenkins.io, the controller for the Jenkins infrastructure test, which pushes all the Jenkins ci.jenkins.io images. But a few Jenkins images, just such as the acceptance test harness. At first sight, I believe that we should use infra ci. My proposal is without thinking in details. It's just because that one is easier to operate. We can in an easier way get information because trusted ci, as its name implies, is a really, really closed controller. And I believe the amount of dependencies and things that we can put on the images here start to be way above what we expect to run on trusted ci.jenkins.io. And it's my proposal to have it on infra ci, where we already have infrastructure level images, which is literally the kind of images you are building, except the infrastructure is only for the tutorials, not for the public infrastructure. Does it make sense, or are there remarks, feedbacks, or counter offers on that one? My proposal is here to operate unless security concerns about the image content. We can revisit the choice of the city, of course. The main takeaway for you, Ashutosh, here is that you will have separated ci and city. That's way different than what you are building with GitHub Action today. Today, you can control when I have a tag, then deploy to the core hub, or when the build happened on master branch, then deploy to the core hub. You have control over this one. You will lose part of that control as soon as we will start the official push because we want to run that on a trusted environment. However, you won't lose ability to run integration tests or any kind of test and build on ci.jenkins.io, which is publicly available. So now that will be you to open Jenkins Infra Desk and we will cover these steps in the Infra Desk. Is that okay for everyone? Be aware that we might have other feedbacks that could absolutely say the opposite of what we just said now, asynchronously on that written issue. That's why we need written communication because overall communication is for brainstorming and asking and answering questions. But that one will serve as an official specification that we can personally use and come back to overtime. So that should answer your question. Is there any other question on that topic? Thank you. No problem. If you forget anything and you want to come back or have any questions further, please ask the question. Start a rough issue with just a bit of context and ask your question there on the threads. No problem. We can answer them. That's not a problem. This can leave. You don't. You are not expected to produce something written once and never touch them. Stefan can confirm that we quite often edit our issues contents. The important part is that we have the question and the discussion persisted. So the future Ashutosh in eight months when he will have to come back saying, why did I make that book a file? You will have a written answer that you can think about. And the community will benefit from that. That's my good. Sometimes you even reopen an issue that you have closed. So no big deal. So everyone does that. That's not a problem. That happened. Is that good for everyone? Cool. Thanks Ashutosh for the work you did on this. That's a great project. I hope you learned a lot and have a bit of a net list. And don't forget to subscribe to our podcast. Don't hesitate. You are free to stay here if you want to continue the meeting, but no one will be mad at you if you stop now and do something else productive. No problem. Your choice. I want to stay for once. No problem. So hello, Oleg. Yeah. Am I audible? Yes. Just in case. So I just wanted to hop on the call after the meeting we had with the CDF. A bit to get you up to speed about what we discussed. Can we just wait for the first pass on the board before discussing that on a recorded session? Because it's recorded? I'm not going to share details on the recorded session. I just said why I joined. If you have a few minutes after the recording, it's all I would need. Yes, no problem. I will have some, so no worries. Okay. So welcome, Oleg. So let's start on the tasks we were about to finish during the last milestone. I'm going to take them on the order of my screen. I will take notes at the same time. Bruno, Stefan, I see that you have closed the issue and changed from nightly bills to weekly ABA bills. Can you give us summary of what you did and the outcome? My job consisted in opening an issue. Stefan, your turn.  We did the upgrade. It was not as easy as we thought but it's done in Pakori Beach and Jenkins-Sanfra and Kubernetes management. That's your lie. That matched all of those changes. So right now we're using the nightly bills of the early ABA of JDK 21. Yeah, so it's early access. I think it's from the second week of August, something like that. We haven't seen any newer version of early access at JDK 21 built from Timurine but frankly we'll get the first official version I think 7 days from now. So that's pretty normal but Stefan made all of this work so we will be able to update to a normal early access version pretty easily I think. What do you think? Yeah, I'm confident. And that's a personal joke. Once the new version will be released you might need to remove the Dash EA suffix Yeah, Dash EA sometimes even EA-beta. We'll see. That's okay. Nothing search everywhere and replace all can't mess up. There is Dash EA also to remove. That will be funny. I trust you will think about that and mention it during the weekly team when it will happen. We have JDK 21 track even an EA version. Cool. Nice job folks. Thanks. Anything else to add on that topic or questions? Stefan, you renewed the VPN serial Yup. Anything else about okay. I didn't forget about the event in the calendar so we have now some trigger that some warning that will trigger two and three weeks before. Cool. So renewed for six months as usual. Why six months and not one year or unlimited? It's a certificate revocation list. So we need this one. We need to control finding this one. Yeah, I know. Ideally that should be renewed every month but that start to be quite the work. So six months is an intermediate that has been discussed years with the security team. We can always challenge that more than six months is out of question. Calendar added. Thanks Stefan for updating the doc. Oh yes, I had it a way to get the new exact exploration that which is not exploration by the way. Next update, thanks. And we renewed the GPG keys for Stefan and Erwe because you changed machine created new GPG keys. So we encoded and we also removed the outdated key removed Olivier or black we removed the outdated. So Olivier if you hear us, you are not able to renew the VPN CRL anymore. It's been two years but yeah now because your GPG key is outdated. So that was the only solution we had to be able to re-encode. If Olivier want access again, he can ask us no problem we can re-encode with a new GPG key. That's all. Thanks Stefan. Is there any question clarification on that topic addition? Okay, so next topic exploration of the digital ocean personal access token. So that one is on May. So these tokens are updated every three months. These are the tokens that allow to create and delete every pieces of resources on the digital ocean cloud architecture which is mostly 30% of the cloud agent of CI Jenkins IU and their caching proxy and soon that will be the home of the new updates by default. So you understand that this infrastructure needs to be secured and regular rotation of the tokens is mandatory. I forgot to add the calendar events last time I renewed this in June. So of course that broke the builds for a few days last weekend so I opened that issue I did on the milestone and changed them. But that time I've created the calendar event so beers on me folks next time. Outside this it's like the CRL nothing to had it's a process we run every three months nothing changes work as usual so unless you have a question or need clarification. Nope, okay. By the way I've closed it after checking that every digital ocean related jobs were working as usual and were fixed. Thanks to helping contributor to make their plugin builds work on windows that was a gradual thing about deletion. The fix was really simple but it had to be done so many thanks for that. The contributor confirmed they were able to proceed. Is there any question on this one? Okay. Just a slight remark building something related to Jenkins on Windows is a struggle in itself so thank you for helping a very brave contributor. Thanks. Now walk in progress. What are the tasks that we weren't able to finish? Some are cross milestones some we didn't have time some are new so let's cover them one after the other and let's continue. First one no action required I thought I removed it from the milestone so I might have added to the milestone here so let me remove it immediately so there is a plugin that will be removed from the plugin Jenkins because it embeds the proprietary dependencies and then there is a long discussion about what will be the solution. There is no action expected for the Jenkins infra team that's why I'm removing it from any milestone. We don't have to track this one. It's more for information. I'm going to take them in my order of priority. The most prior today's artifactory bond with reduction project. We are officially not mirroring Apache Central Maven since last Friday. Hi. Mark asked for a g-frog and received the logs until Thursday or Friday. So now we are tracking logs. The idea is that in two weeks we will start to see a positive impact. Why two weeks? Because we have cleared the whole cache system so now we have to refill that cache which involved downloading a lot of dependencies. Only once but we have to download them. By the way there is an issue on the Maven HPI test plugin and I need help. I understood the problem so I partially understand it and I need help to drive this to completion. The integration test of that plugin are broken since we started to fine tune settings XML. I need help because it involves a lot of all Jenkins decision that were made in 2015 with Maven whatever version. The initial idea that Basil also pushed on was to remove the old plugin in charge of trying to proxy every dependency so that each integration test scenario doesn't download the dependency each time. However, that plugin does not resolve properly the mirrors and everything. So it was failing on CI Jenkins but not on contributor individual laptops. My problem is whatever I tried when playing with settings XML use for this test. I cannot reproduce on my machine the error I see on CI Jenkins even with the same Maven setup. So there is something I'm missing maybe obvious but I need help from let's say specialists to give this to completion. If I can't find anything on my own I will have directly Basil Daniel the usual every Jenkins contributor I could find from the past seven years that was just a note anyone interested you can look on the issue. It's not blocking artifactory because the thing is that we cannot afford going back or GFrog will be really mad at that problem but we can't afford not having integration test working for the Maven HPI plugin so we need to solve that second issue as soon as possible. Is there any question for this one? Okay. I'm putting now I am 64 so Stefan can describe his work while I take notes. You want to pose in fact. I try to create an exhaustive list and in fact I missed a lot so I have to work again on that exhaustive list which is not exhaustive but the system shared pipeline library that built our images is now fully functional even with the legacy part for windows and the new one even when we specify path and specific arguments so I switch to migrate a few more to IRM 64 for the public K8S cluster and we it's another issue I merged two issues in my brain because we also work on avoiding IRM on the same node. I did prepare everything for wiki and for account app so account app should soon be ready to IRM 64 Okay, cool. Thanks. So that mean we have the list then let's process I need to merge but my proposal is that we start with what is in your list so you don't need to spend more time on other images is that okay for you and we proceed and that mean we can also hand over tasks or move if there are some that are easy we can move them to new contributors if they're interested Great. Is that okay for you? Yes but we need to add the fact that we need the CLI for IRM to make sure that we got the condition to make sure we have the IRM right now it's not written anywhere I did in the issue not absolutely exhaustive but a good start warning about update CLI must be aware of the IRM 64 images and check both IRM and do you agree Stefan that it's an opportunity for us for each of the images to push them no no no for each of these images to properly check and fix the life cycle meaning we should not use latest anymore on any of the images and instead use our automatic tagging deployment system yes to pin them yes I have to switch images with an SH that we build to send their automatic deployment instead of latest or manual or weekly updates and they say friends don't let friends use latest so we were using kind of latest that's something we checked with Stefan because we had to update some images with packages regularly because you don't want a CV so first problem is that some of our images held up for not naming it hasn't been changed in two years which is catastrophic from my point of view and the process should have been update CLI or whatever tool say oh this week what is the SH of the latest version you can surface to the operators to the production operators that oh there is a new SH there is a change so you have to find a change in that case it was a SH for numerous reasons including no solution were able to to get the source of reference for these images because we don't have tags so the pattern for building these images is the same as what Ashutosh did for the GSOC we were building a main branch and we were overriding the latest tag and then we had that detection system that say oh there is a new SH that's not bad or wrong that's just how it used to be which now with the RM64 problem we need to switch to tags because when we deploy a tags version we defer to the Kubernetes cluster the choice of oh I'm running an RM64 I want to manifest for the RM64 for that tag the image otherwise if we don't use SHH checksum if you want to stay strict that mean we need to update all of our Elm chart to say oh the Elm chart must know based on the node selector oh I'm going to run an RM64 here is the checksum for the RM64 or Intel or then my checksum is there which mean updates I need to keep track of both SHH not a problem it's just we have to know which strategy and today we are enough and we have enough money to sustain a version based with tags because that would be a way for us to avoid dormant images like LDAP does make sense yes I think we also need to spend a few minutes few few hours to improve the chart pipeline library to tag and push in one pass and not having main tag but we need to speak about that yep speed up the docker image library to create tag to create push tags at the same time for both SHH and docker instead of running additional builds so Stefan can I ask you for the upcoming milestone to open an issue describing the expectation here you don't have to find a solution on the code you just have to describe the high-level workflow and I propose the following reason that will decrease by half the number of builds we do on infra-ci for docker images or at least 25% but in any case it will be less credit spend on Azure yes, agreed issue to create right thank you why is to decrease build costs from infra.ci is there any question about RM64 did I miss something on my notes that you said Stefan nope, can we speak about the anti-fint here right after this one because it's matching we are let's go with high availability of the services on the public cluster thank you so what did you do on this one not much I just prepared a request to make sure that any kind of application that have more than one a replica account got this replica account on multiple different nodes and of course I started with the RM64 because they are the one that are most more probability to get that because there is a few nodes and that means that Kubernetes tends to spawn all the replica on the same node cool and I was able to review your pull request and validate on a local cluster that it works and the next step is to deploy and see how it behaves on real life with a.k.a. auto-skiller because that one cannot be tested except in production so we might bring down the wiki for a few minutes but better doing it in a controlled way so that means you need to decide an operation time open a status check in SAIO and then we can merge and see that one that shouldn't take long I was about to do that right away after the meeting so I will have to wait if you trust that it won't break the system no problem otherwise there isn't no problem which means then we need a list that's the new but that should be almost the same list as the one you did for the RM64 a list of tasks to apply this then we can create issues and mark them as a newbie and then that means you will be able to drive the newbies if you have new contributor that want to try opening pull request and contributing you can drive them give them feedbacks we have Octoberfest coming while you can spend your brain on something else more valuable than drinking the beer for instance is that clear perfect no more question okay next step blah blah blah I'm going to report for Hervé about updates Genkin Sayo migrated to another cloud so problem statement update Genkin Sayo run on AWS on whole virtual machine which is not even idea available that's a really important service that's the most important service of the world infrastructure as I would say the goal is to use mirror redirector system so Hervé was able to build an HTTP redirector which is running on Azure and now is started to parallel tracks starting a mirror on digital ocean using the same chart as we have on Azure cluster so we use the same bricks that will be the default mirror that will be hosted in Frankfurt then in parallel is working with Cloudflare to start a sponsor chip so they will host mirrors of this one in the form of S3 buckets the goal is to have one on US East and one in Azure event so the proposed timeline will be having a brown out before end of September to try last week of September unless we discover something complicated to move the update tank inside a domain name for one hour to the new one on Azure and see the result of course we will test with controller before that and we will ask from security and usual contributors but the whole idea is that we have redirect so we can continue using redirects that will remove from 5 to 7K of bandwidth billing from my AWS worst case if everything migrate to digital ocean so digital ocean primary mirror is being built worst case billing scenario that should be around 700 bucks per month which is 10 times less than in AWS so even the worst case that would be easier for us if we cannot sustain both CI GenKinsayo builds and update GenKinsayo on digital ocean we will move CI GenKinsayo somewhere else or find new new sponsors and so airway has started contacted digital ocean or is going to to increase or move the the sponsorship here the goal is to leverage the impact of moving that service to Azure on Azure itself loadflare answered and is evaluating sponsorship for US and Azure worst case we have a box for we can sustain one year of mirror for one year on US as secondary fallback we have a clean deployment of LM charts so the next step for airway will be to continue installing the mirror on digital ocean and then deploy the update center Gison index on all the mirrors at the same time so that will be done by trusted CI GenKinsayo every 15 minutes so that will be the next big step is there any question or clarification on that the most important topic one, two, three, okay so now, now, now I got a few SSL certificate expires in 25 days so Mark thanks Mark our best monitoring system ever detect that the 5th of October the certificate for this free domain will expire so after a quick research when we moved on migrated AKS clusters a change was propagated that means these domains are the three of them are pointing to our cluster which redirect them to fast leak for WW GenKinsayo CDNIs the version of the website the thing is we still need to have TLS connection before the redirection happening and the culprit is there is a problem with the certificate renewal inside our AKS cluster I try to give a summary here but we need to remove one certificate let's include setup from Fastly and remove one ingress domain from our AKS cluster and then renew the certificate that should solve the problem once for all the main reason is because we have one ingress with four or five hostname which mean one certificate with suggestions alternative names and one of the domain names WW GenKinsayo is failing which make the world certificate renewal to fail and why is it failing because WW GenKinsayo is managed by Fastly themselves let's encrypt include it I think every and I missed that cross domains due to how fast it works so that should be the culprit so we are working on this one for the upcoming milestone I'm taking it by default any question? next one is we had unexpected slow times on some bomb steps so basically the bomb is a way to track GenKins bugs and there is that long running bugs when you have a lot of concurrent agents running a lot of concurrent pipeline steps that make it slow down somehow it's really hard to reproduce and to track but when you have 300 or 400 parallelized steps which is our case for the bomb steps some SH steps and now as Mark discovers some stash and unstash steps as well takes minutes when they should take seconds so for instance seven minutes to run a curl head on one of the ACP the SH process takes less than one second but then it takes seven minutes for GenKins to retrieve the state, close the connection and do whatever they do internally so due to that JSC helped us and we enabled on some of our controllers a new startup flag which changed the way the pipeline cps system, the durable task when you restart controller pipelines can resume after the restart the way that mechanism works it changes its behavior I don't know the low level details or I don't want to share them now it's public but we are trying that new mode now that should have a great impact so now we are going to watch and measure the impact on the bomb bits is there any question on this one okay so now I'm going to track this with JSC this one has been removed Gira login page on the left status mark and high we didn't have time on this one Stefan, status on matomo yes there is two different parts there is the MySQL instance that I started to create as code infrastructure as code so we did merge the network part and now we are I have a request in draft that need some discussion and review that should provide MySQL managed instance to use as for the rest the image I've done nothing yet on it okay so new subnet draft for the terraform manage MySQL instance so we need to discuss is that correct yes normal work yet on the docker image no you you already started to clean right I I think I did start something two or three weeks ago but not this week but normal work yet okay yeah I think something like I am 64 that's what I started cool next certain email from SPF I didn't have time to look will you have time to look on this one Stefan honestly or do we move it to backlog you mean by myself the problem will probably be to understand where those mail are coming and where is sending them and will circle but that's the problem that's just a question do you feel you will have time given I'm off and there is most probably he'll for one or a few days I will try I will try okay that's something I like my let me have some time to spend on we'll try to find some things just a word about Jenkins Infrapecary Mage from our new contributor was not able to visit us this week he started to work on playwright artifacts because we discover on the CI Jenkins IO agent whether container or virtual machines but our Linux we had unwanted artifacts mainly package, json and dot modules and that wasn't a blocker he found that it's because of the sanity check because we check that given the dependencies we installed during the provisioning then we should be able to install playwright test and run a command by installing that test framework and the associated command it will embed the tools and its dependencies creating these leftovers so we started a discussion do we want to have that tool as a tool and we need to track it instead of using latest or do you want to only keep removing them as part of the sanity check so Stefan I will let you continue the discussion from there since you initially implemented a playwright I try to extract information from my memory and put them on issue but I need you to help the contributor if you can't we will brainstorm with him next week if you don't have time Aida I didn't even remember that but yes I will try to move my brain as I said on the message recommendation is to only clean up artifacts after the sanity check and I believe they started to work on Jenkins CLA documentation as Mark gave them but that's more non-infrastructure topic so thanks for that work non-infrastopic so just mention because we have someone else helping us so that's very nice and that's all for me sorry I kept you really long do you have other topics you want to have okay I haven't seen a new triage issue so that mean we can close so quick reminder I'm off Thursday and Friday shouldn't have an impact but yeah at least let's work done but I trust Stefan will do a lot he already did and we'll see if he's going better any one last word before I close the recording no okay I'm stopping screen share thanks for everyone I'm stopping recording see you next week for people watching the records


From London, England, extracting the signal from the noise. It's theCUBE, Cover, Discover 2015. Brought to you by Hewlett Packard Enterprise. Now your hosts, John Furrier and Dave Vellante. Okay, welcome back everyone. We are here live in London, England for HPE Discover, HPE Enterprise's first event as a newly split company from the HPE Consumer HPE Inc. This is theCUBE's SiliconANGLE's flagship program. We go out to the events and extract the signal from the noise. I'm John Furrier, the founder of SiliconANGLE. I'm John Michaels, Dave Vellante, co-founder of Wikibon.com. Our next guest is George Ferguson with HPE Enterprise, workplace productivity, welcome to theCUBE. Thank you very much. So you are part of the pillar, one of the four transformation area pillars, workplace productivity. This is really about user experience apps, agile programming, but enhancing the overall environment around just how people interface with technology. Absolutely. A little DevOps feel to it. There's a lot of DevOps involved at this show. A lot of stuff going on around this. Can you unpack what this means? Because at first glance, it's the future of work. It's also productivity. Is it VDI? All these things are kind of now back on the table. How do we think about this? Okay, let's talk about it from several different aspects. One aspect, intelligent workplace. How do users get support when they need help? Whether that is setting up an email environment, getting assistance when something doesn't work with their PC. They're very traditional ways and there are very next generation ways of doing so. We'll explore that in a minute. Communications and collaboration. Yes, encompassing email, but also definitely Skype for business. Different ways of communicating, presence, video. Very cost effective means of communicating that may not have been available a few years ago. Next area, wireless workplace. Can I move around the workplace? Do I have to stay within my cubicle to maintain connectivity? Can I walk anywhere across the campus, through an arena, through a museum? And finally, digital user experience. The seamless bringing together of the physical world and the virtual world. So all of those are part of enabling workplace productivity. It is a big area, you're right. And as an IT CIO and or CXO in a company, obviously now the focus is on growth, revenue opportunities. You're seeing the tsunami of new creative lightweight applications built on the cloud or on prem. But it kind of turns upside down the older notions. We're talking with Elaine or Elaine earlier about the servers, how these to be high-end mission critical servers and then general servers. Now that's completely changed. You have general purpose, mature servers and then specialized kind of use cases. So how does that translate into some of the older paradigms like unified communications presence? I mean, that's an older concept been around for a long time, but that ties into mobility. That ties into voice communication, videos conferencing, social media, omni-channel connections. How does a CIO figure this out? What are some of the things that you guys are doing with customers to help them make sense of these older patterns and make them more modern? Fundamentally, the business needs stay the same. I need to improve my employee engagement. I need to improve my employee productivity and I need to better instill loyalty among my customers. Now there are some definitely some new ways of doing that. As we talk about instilling customer loyalty, there's an opportunity to reach out to customers in ways that simply were not available before to bring together the experience they have on their mobile device with the experience that they're having in a brick and mortar store, walking through a museum, attending a sports game. All of those can be looked at in new ways because of the technologies that are developed. So how do you actually measure workplace productivity? Is it revenue per employee? Is it to depend on the industry? I mean, how do you measure it? Or how do your customers measure it more importantly? It depends on the various different aspects of it. Certainly, minutes saved is simply one measure that we look at for our implementation of Skype for Business within Hewlett-Packard, combined Hewlett-Packard Enterprise and HP Inc. We saved 18 million minutes over one year with our deployment. That was a time that was saved because in the past we needed to go through and key out. Here's how we connect into this conference line. Here's the pass codes. Now we can do that with one click. That's actual savings in terms of productivity. Now, the other aspect to it is what's the cost of doing that? And that's the question people ask. People want to have certainly productivity gains for their employees, but they say, oh, but if this is going to cost me a lot, can I even afford to do that? Yeah, how do people pay for it? And the interesting answer is that in most cases, we can achieve the productivity gains and at the same time, cut costs. That's an amazing sort of thing to most people. But for those same savings that we're talking about with Skype for Business, savings can be $1.5 million according to Forester for just 1,000 users. Per year. So through a variety of different means, automation, self-service, we can actually make savings and improve productivity. It's a great combination if you can do both together. But there's still going to be a capital outlay. Is that fair? We can use HPE financing to basically eliminate the capital outlay. We can use consumption models, whether through finance or through the cloud, to actually make this, okay, you spend the money as you use it as opposed to the upfront expenditure. Sounds too good to be true. It does. Now what about this notion? You mentioned a digital user experience before. What do you mean by that? Okay, the real idea is we would like to have a seamless experience as people move back and forth between the virtual world, typically on mobile devices and the physical world. I'll give you an example of in the negative first and then in the positive. Back a few months ago, my wife and I went into a major department store. She'd seen an online ad for a particular, I think a blouse or something and walked into the store. Oh no, we don't have that. That you have to get through our dot-com interface. That is not a seamless experience. What we instead would like to have is an experience where as I walk into the store, it knows that yes, I was searching for something on their website. It'll actually direct me to where that particular piece of clothing is and it will also give me the same discount that I could have gotten on the dot-com site. That's a seamless experience. A seamless experience is presented today during the main stage event. People walking into the Museum of Fine Arts of San Fernando in Madrid, Spain. They can go and use their smartphone to say, okay, I'd like to take a tour for one hour. That's all I have. And by the way, I have a 10-year-old child that's going on this with me. The app can go and tell you, okay, here's the path that you're going to take. We're going to skip you over these areas. We're going to have you go to these areas. Step by step, it'll give me directions. As I view a particular sculpture or painting, it'll tell me about it. That's a seamless experience between both the virtual world and the physical world. That's where there's excitement. That's super cool. I mean, that's real cool stuff. And obviously Oculus Rift and these new VR environments are going to create that user experience. And a lot of developers are looking at this as an opportunity. This comes back down to the composable mission, which I love, by the way, love the word composable. Reminds me of service-oriented architectures with now with microservices. A lot of great opportunities. So developers are going crazy for Oculus Rift. These new experiences. Talk about the role of the developer in all this. Because at the end of the day, this is the opportunity to kind of bring it all together. We have a variety of different tools for developers, App Pulse Mobile, Mobile Center, so forth, to help them first in the actual development exercise, but then in terms of measuring the performance, testing of it, handling the security aspects of the mobile apps. Each different aspect of the application development lifecycle is assisted with various tools from HPE Software. And what traction do you guys have with developers? Can you share any anecdotal use cases of how you guys are onboarding developers, some successes you've had? Can you share any data on how the workplace stuff is kind of this, the new millennials? For instance, they love video, right? So we see that, obviously, here in the queue. But what are some of the things that you're seeing developers uptake on in terms of this new workplace environment? I would say there's a great appreciation for the, particularly for the aspects of the performance of these things. That's where I would say there's the biggest uptake. If you have to wait three seconds for a mobile app to respond, you're going to go to a different mobile app. We have the tools to help a developer to see, oh, this is what I'm really going to get with this app. This is what's going to make it successful. Yes, it's, of course, the user experience, is not just what you see on the screen. It's the performance that you actually have.  No, please. What's the top conversations you're having with customers around this area? Because this is an area, again, it's exploding. A lot of creativity, new solutions are hitting the market. What are the top three conversations that you're having with customers around this? As they say, help me, I need to figure this out because I got these young guys coming into the audience from the user base coming in to work for us and they don't want to have three phones. They don't want to have a desktop desktop. They want to have a seamless experience like Snapchat, Instagram. Right, right. And so, what do you, how do you help me? Well, the way you presented that was excellent. That is oftentimes how we start the discussion. Users are used to a consumer experience with their smartphone. They're used to being able to support themselves in getting applications, support themselves in terms of getting assistance. They're used to operating system updates that just flow to their smartphone. It's a consumer experience. And then they go into the enterprise and, oh no, well we can't do that. That'll happen in six months. That'll happen a year from now. Oh, this application, well it's what we use. What we're trying to do is to reduce that gap between the consumer experience and the enterprise experience. And we can do a great deal for that. It's critical for IT to reduce that gap. If not, what users will do is go around IT. Yeah, the shadow IT market is a great R&D opportunity. It is, and unfortunately it causes huge security and compliance issues that IT does care about. So we've got to provide the means to help them have the security and compliance they want and get the consumer experience. I'm a CIO, George, help me with my problem I have. I want to close that gap. What do I do? I would talk through the same four areas we talked about earlier. For starters, I would talk about, okay, what is the provisioning and support experience for users? In the past, what that has typically meant is a very ugly phone tree with somebody who may or may not speak your language very well. Instead, we can make that and so come into a walk-in support center. It might seem counter-intuitive, but we can actually do this at less cost. Walking in, users love it. HP has used it. Hula Packard Enterprise now uses it heavily. We're deploying these for our customers. Now this works great if you have a couple of thousand employees at a site. What if you don't? In that case, we can provide you a video virtual agent that you can talk to, have locker systems that the video agent can go and unlock to give you, okay, here's a loaner PC while we're working on your PC. Here's this help. Yes, I can go ahead and take control of your PC. We're demonstrating this over in our Naval Workplace Productivity Area with a video agent talking to people. We can handle a number of sites in this way in a very automated, cost-effective means. That would be one example, and it's only one example. So what do you actually sell in that example? Do you sell a solution to a specific problem that involves infrastructure hardware and software, or you bring in a partner network? Can you describe the transaction if you will? How does HP make money at that? Two different places. First, we provide software, so we certainly provide software like Propel that can help provide that self-service experience, make it very easy for users to get the information that they need. Service-anywhere, similarly can provide assistance for that self-provisioning sort of measure. Now, embedding that software, so some companies say, okay, the software solves my needs, I'll handle it myself. Great, if that's where they want to stop, that's okay. If they would like Hewlett Packard Enterprise to provide the full support experience using that software, we can do that on a managed basis, and so we actually provide them the workplace support services using those various different pieces, and there will be different means. Certainly, sometimes it'll be totally automated, sometimes it'll be self-service. Yes, we will go back and have real people that you can work with if that's what you need, but we're going to try to develop a variety of different means to interact with the user so that it's less expensive and yet provides that superior user experience. So you mentioned workplace support. Sometimes that has negative connotations, you know, help desk workplace support. How are you addressing that? All I can tell you is, here's my personal experience. Back a few years ago, in a cost-cutting frenzy, yes, I used to go through that experience. It was bad, I know it as a user. Now, if I have a problem, I walk over to our walk-in support center, yes, I do my badge in to show, okay, here's my issue, pick from a couple of options. Typically, I have somebody helping me within probably about two minutes. It's very genius-bar-like. It's like a genius-bar, yes. And wow, it gets fixed. I'm a happy user. I'm back being productive again, and yet we managed to reduce the support staff to do that by about 75%, and yet made the users a lot more happy. We go and actually help companies set up that sort of genius-bar or walk-in support center, and can then- That's a big solution right there. That is a big solution, yes. Real aspirin and a vitamin at the same time. Gives them a little relief, but also sets the table for the next generation. I like that. Aspirin and a vitamin, I think I need to take that line. I didn't say vaccine, but that seems when you really need some grip, and the paddles come out later. George, thanks for sharing. One more question. So you laughed before when I said revenue per employee, but ultimately doesn't, isn't that the metric that gets affected? Because if you're, people generally don't want to fire people, but they want to redeploy them. You always hear that. No, maybe they do. Absolutely. So isn't that driving more revenue per headcount? I mean productivity in the U.S. anyways, going through the roof. No, certainly as I... Good point. If I have time savings in terms of doing these administrative tasks, I can put more time into my job. And yes, that directly then relates there to revenue growth. That's how you pay for it. So that's how you pay for it. From a cost center to a profit center. Absolutely. George, thanks so much for coming to theCUBE. Really appreciate it. Thanks for sharing your insight here on theCUBE. We're getting all the data. That's where we're data-driven media. That's theCUBE. We are one of our transformation pillars of data-driven social media. Well, thanks for sharing the insight. We'll be back with more live coverage from London, day one of HPE Discover after this short break.


Thank you, Mr. Speaker. Mr. Speaker, I'm pleased to support the motion that is before this House. The boring of this very important amount for the completion of the St. Jude Hospital and the restoration of the George Odlam Stadium. But Mr. Speaker, I just want to establish the context in which I will make my brief contribution. Mr. Speaker, the 9th of September was the date of the fire. And the 9th of September happens to be Josima's my son's birthday. So there is a connection for me personally when I speak of the 9th of September. And while I was in Martinick on the weekend speaking about seeing about my health on the 9th of September, while grieving my son, I realized that we were also reflecting on that great day when on that sad day when we lost a very important facility in the south of the island. Mr. Speaker, there's another reason why I rise to speak on this issue. Mr. Speaker, there are many subjects or themes that go through this Parliament. There are legal themes. Of course, a lot of the discussions surrounding the debate in Parliament. And I'm not a lawyer. There are economic themes. I'm not a student of economy. I pride and I love the area of social development, but I've studied at both the first level of a degree and master's level as a quantity surveyor. And whether it was, and one of the things, Mr. Speaker, that always, you know, challenged me is that I was trained by a West African in Jamaica, a quantity surveyor at the highest level, who said that when you are preparing a payment certificate, if it is for your enemy, forget your emotions and prepare a payment certificate as accurate as it should be. And Mr. Bochi, my then lecturer at CAST, then the University of Technology, really impacted my life, my professional life in dealing with the issues of construction. Mr. Speaker, when we won the election, even before we won the election, while being in opposition, I was the leader of the opposition representing the Prime Minister then. I continued asking the question in the Senate, sorry. I continue asking the question to the then government on the reports associated with St. Jude, because clearly for me, as a professional, there was something missing in the discussion. Mr. Speaker, for anybody who is a professional, it would have been obvious it would have been as daylight what was wrong about St. Jude. Mr. Speaker, any serious government, any government who is up to the game would not aim back on this continued project as complicated as a hospital after you, after they have assumed office. To do the following, demobilize an established contractor to get rid and store materials that you no longer have in use, that has been paid for. To lose supply arrangements where suppliers are now gonna drift, you don't even know where they were. Fire the managers of the project, you'll lose institutional memory. To do all of that, why would you do this? Even when in private contract where some contractor is building a home and there are issues, they go through arbitration to resolve the issues than to abandon totally the procurement arrangement that has been put in place to supply a particular project. Mr. Speaker, millions of dollars at stake and of course the minister of finance gave an account where somebody had to tell him, sir I am holding $1.8 million for you and had the person not made that statement, he would know nothing about it. And I asked myself how many more persons like him have come forward to state that they are holding in trust because there were many deposits made on account and there were many suppliers, subcontractors suppliers who probably know nothing of. Why would a government just get rid of the entire project management team with institutional memory and have bought a project at this stage? Why? So Mr. Speaker, I used the opportunity and I decided to go into details and I paid attention over the years to some of the statements made by the then minister. Mr. Speaker, I can tell you that based on a little of what I've seen, of what I've heard, the former government had very little consideration for any procurement strategy. There was very little appreciation for a procurement approach to learn anything that they intended. They were operating in the dark and somebody else was speaking to the government and not anybody who had command of the program or what they wanted to give to Senbusha. So for example, Mr. Speaker, let's start by saying that you need a house because you're going to get married and you want to move in within six months. Of course your procurement approach would have reflected that. You will probably go into an arrangement where the designer, you go into a bolt arrangement where you have a fast-track arrangement but why would you have bought an ongoing project and decide to build a regular frame? I do not want, I will call it a box because the box is an appropriate term and the first time I heard the box was at school because it is the cheapest form of arrangement when you're building a building based on the fact that it is a regular frame. Now some people take offense to the box but the box is a right term to use in the context of the type of construction but in the in the more technical term they would say a regular frame meaning that it's rectangular. That's what it means. It is the fastest and should be the most economical form of construction. You can move fast. But Mr. Speaker, somebody said that they could land a hospital in St. Lucia within a short time frame and the politicians then the government believed it and they communicated it to the public and Mr. Speaker, they failed. They were so convinced that they would have delivered the hospital within that time frame. They said eat it, don't. Please do not vote us back in. To make such a statement if you are not a professional it's because somebody convinces you that you can be that it can be delivered. Mr. Speaker based on what I've studied it is believed that the fastest form of construction is a regular frame a box. As a matter of fact most of the the the claddings are already on the market you can purchase them and it is easy to retrofit. But Mr. Speaker, I can tell you on that day I visited as a senator the St. Jude site. Everything was there to see as a major error. I asked the engineer on the site but is it a single-story building? He said no it's a two-story. I said how could it be a two-story because you cast the first floor but there's no ground floor. Ah he explained. If you're building a two-story building with a ground floor and a first floor why is it you cast the first floor and the concrete roof but there's no ground floor. He said so we have not received approval for the drawings for the services. So there were no approval of the building while it was being constructed. That is not a lie the engineer on site said that. I don't know why the minister is trying to the the former prime minister is trying to make a case and then the engineer knowing that I am a quantity surveyor he said Mr. Henry I'm just looking here. I said but that's crazy the mechanical drawings for the plumbing for the electricals for waste disposal and when I discuss with him waste disposal because one of the most challenging areas of a hospital is the disposal of waste. Waste from surgery. Blood. The gouges. Where do you dispose of it? How do you get that into the special arrangement to get it down into those treatment plan? That was not considered when the first floor in the roof was built. They were at the ministry still working on the drawings. Mr. Speaker there wasn't one conduit in the building at the time not one piece of conduit to pass electrical wires not to run pipes for drainage. Somebody told the government then that look here you can build a hospital within that time frame and they decided to bring back on this. Mr. Speaker the government that I'm part of we do not subscribe to any of this. As a matter of fact the prime minister I'm a quantity surveyor in the in the cabinet I do not speak in the cabinet as a quantity surveyor. I speak as a parliamentarian. I speak as an officer in cabinet and he subscribed himself to professionals outside to guide him and I'm happy that the prime minister use professionals to advise on the way forward. I had my bias from day one. As a professional I was influenced that we should not continue this regular frame structure it will not see the delight. It won't see the light of day. As a matter of fact Mr. Speaker when I entered the building I laughed because it was the first time in my life I saw the quantity of drywall stored in any one place. Mr. Speaker this place would have been the home of drywall because if you look at what is what you see on television aluminum frames to serve as the studs and to put drywall partition and to set up rooms throughout the entire hospital. That is what was there but there was also a major problem. I brought to the attention he agreed with me. The concrete roof didn't have the membrane and I'm happy you could see that the former prime minister was told something. The membrane the membrane might be asphalt they usually put on a concrete roof you put asphalt or what you call tar. You put some pebbles and you put some some some membrane on top of it to be the roof. This is the most troubling thing in a tropical country. Up to now we cannot resolve the waterfront buildings and leaks. When it's hot the membrane melt below it melts the asphalt and you have leaks and you cannot determine whether leaks are sometimes you have to rip off everything and do it over. The irisomans school I worked on up to now we have issues with it there are so many buildings in the region with concrete roof we have not resolved. We believe that it's a safer way. We believe that we're preparing a platform for helicopter to learn but a major problem as it relates to the cages. Then I saw little holes on top of the roof I asked him was that he said this is how we drain rainwater.  This was he said well we need to have a certain strength of the roof to withstand when the helicopter land. I said but you have patience on below. This is not how laminar is designed. That was not approved by planning. I didn't join politics to do a TMT but if I sit as a quantity surveyor in a cabinet room in parliament and say nothing I believe I would be doing a disservice to send lotions because they gave me a scholarship to go to school. Mr. Speaker I believe today what you heard from the from the former prime minister is misleading. As a matter of fact while in opposition I asked for the report the very report that was hidden for five years five years and the report that they invested one million dollars into. I tried to hold my breath on it but most times I asked why did they pay for the report when all I saw was the draft report after 12 weeks you supposed to receive the final report and then pay but why did they pay for the draft report without receiving the final report after consultancy. I do not know if there is a final report but if there is up to now we have not seen the final report because what we are making reference to is the draft report but the draft report when we came to the to the to the parliament as a senator to make a document of the house again they resisted why then but the report said something that's very interesting. The member of the castries office you are aware of the motion before us. Absolutely Mr. Speaker. I am trusting at some point your contribution would at least collide. Mr. Speaker I'm heading there I'm about to land on that borrowing. I'm about to land on the borrowing. That's small the small one I went to Matt McCormick. Yes Mr. Speaker. Mr. Speaker the issue of this building and everything that we encountered and the discussion that led to where we are today is that the building was unsuitable. The corridors were narrow and they aborted this project wasted money and today this government must find the resources to complete it but more importantly Mr. Speaker one would ask where's the justification for the resources borrowing one of course the Minister of Finance of course alluded to it about the fact that we're getting some in very encouraging terms of borrowing 2% five years grace but Mr. Speaker it's important the five years grace because most times when we speak of buildings we speak of just the capital outlet to establish the building but we do not speak of the life cost of the building the life of the building and when someone gives you five years grace what it means that within that building that you've loaned for you now have you are owning that building for five years without paying a cent for it. Mr. Speaker the building that was abandoned for three years could have been completed for far more for less had this had the previous administration not abandoned the building and allow bats, rodents, bush to take over and nature have a way once you if you go on holidays too long and you leave your home in Carp estate the trees will not stay away the plants will invite themselves that's the nature of that's how nature operates so they abandoned the building and when I visited the building the bats the trees and everything had actually taken over but not only that Mr. Speaker materials based on Shanta Kings report in excess of over 40 million 40 million dollars the procurement arrangement was in place. Why do you abandon a supply arrangement on a project as complicated like this Mr. Speaker? Mr. Speaker when you prepare in a procurement plan for a project like that very complicated is how you're going to get the equipment and specialized arrangement for a hospital these things are not available at Johnson's and Johnson's they're not available at the at the warehouses and business places in castries these are specialized equipment the tiles must be non-static you do not get them in the the ceramic and porcelain tiles in the warehouse or there you must make special arrangement to get them you had all of this they must be stored at certain temperatures they were stored you went into the place where they were stored you read them up as if the person was robbing us and up to now we do not know what has happened to them I do not know and then you're going to start a new building Mr. Speaker that makes no sense I am happy that this government has a different approach and I believe that the prime minister and the cabinet of ministers one of the things I recognize about them it looked like they sing in more of Bujuban than song action not a bag of mouth you understand we are not in for the long-term it's action we are about so our approach to St. Jude Hospital was basically what's the problem and what must we do about it and therefore while they were matching and promoting and doing all the demonstration work had started on St. Jude long work involved the assessment one of the things that we also did was whatever materials that were left there to do a suitability fit for purpose assessment because we know that these materials because they are dead doesn't mean that they are suitable because you have to do fit for purpose assessment these materials had to be stored under certain conditions they were not the persons were responsible paid no attention to this they went and they took materials and they put them in containers they store them under any condition and it right now have no value we cannot use them and that is why the amount that we have to spend today is a lot more Mr. Speaker on buildings like a hospital you would find that the cost of materials is far higher than the amount you spend in labor and if you have 30 million dollars of material store it means that with approximately 20 million you can create value of over 70 million dollars so when it is said that we need 70 million or 80 million dollars to complete the hospital the amount of materials there and the materials added to it and the profit and preliminaries would bring it up to 80 million that is significant in the discussion but when you lose 40 million dollars of material of course it means that you now need to get 80 million dollars to complete the hospital that's what it means in quantitative terms because material cost is only a fraction of the total cost also significant for us to understand is mobilization Mr. Speaker in mobilization when a gentleman has he's taken a low-boy and he put his excavator his excavator machine on it and he takes it to a site and he charges you two thousand dollars once the equipment is there he can now excavate for you at 120 a day you do not have to pay for the mobilization of the equipment when a contractor is already mobilized at the cost of over half a million dollars and you stop him it means that a contractor you're bringing in at this time you now need to spend another half a million dollars to mobilize him on a project with all which was already on trade you do not change the rider of a horse in the middle of a race and I thought this former government was so much expertise in horses would have known that you understand I would have known that what they did they changed the rider of the horse and actually start all over Mr. Speaker I will keep this brief because I suspect there'll be another time when we need to go into further details for St. Jude Hospital but I would tell you I am happy that our government spent no time to talk much but his action not a bag of mouth thank you very much Mr. Speaker