So, welcome everyone to this parallel session on building resilient e-addisar systems in Sierra Leone. My name is Johan Sade from the University of Oslo and I'll moderate this session as as far as practically possible. We have two presenters for this session. I will just briefly introduce them before I'll hand over the floor to the presenters. So first we have Richard Magoba from Affinett. She's a public health informatics specialist with four years experience in disease surveillance information system and she has been involved in setting up e-addisar system in Sierra Leone. The second presenter is Kalle Hedberg from South Africa. As you can see, he has a long experience in the field of HIS and also a pilot known as a pilot in his network. So without much more ado, I'll give the word to Bridget and we have around 30 minutes for Bridget and Kalle to present. Any questions if you can write those in the community of practice, we will bring them up after the presentations. So Bridget, please you can go ahead and share your presentation now. Okay, thank you Johan. I'll share my screen. Can everyone see my screen? Yes. Good afternoon. Good morning. I'm Bridget Magoba from Affinett, Sierra Leone. I'm presenting on the electronic aspect surveillance system in Sierra Leone. Just to give a brief, in Sierra Leone, we currently have the weekly surveillance reporting system which is aggregate, then also the case-based electronic disease surveillance system. So I'm going to mainly focus on the case-based disease surveillance system. So in Sierra Leone, the Ministry of Health, with its partners CDC, HIS South Africa, Affinett have set up the electronic case-based surveillance system which reports 20 plus notifiable diseases. So this was set up since 2018 with support from HIS South Africa as Kalle will also add on. In the initial stages, we developed the enrolment. It is a tracker program, so it has several stages as we're going to see the next slides. Then the system went through several revisions through partners WHO and CDC with the Ministry of Health, another directorate of health security and emergencies, and then it was piloted in four districts out of 16. In February when COVID outbreak came up, we expanded on the very existing case-based surveillance system that we had and incorporated COVID-19 as one of the notifiable medical conditions in the country. So in Sierra Leone, we did not develop a new instance for COVID-19. We built on what was already existing. So during the pandemic, the Ministry of Health relied on the case-based surveillance for information management, which was rolled out in the 16 districts and it was what is being reported for report for COVID-19 data up to date. Currently, we are expanding the system to the other districts for the other notifiable diseases besides COVID-19. So this was a lesson from the Ebola outbreak, where we needed to use the existing applications to build on other new information systems or other new diseases that come up. So as I mentioned earlier, it's a tracker program that has different stages. One is the case registration, which is the enrollment, and this is mainly captured by the health facility staff using mobile devices. In Sierra Leone, we are using tablets. During the weekly disease surveillance reporting, which was rolled out the entire country, each health facility received a tablet for reporting. So with the electronic space disease surveillance, it was easier for us to use the existing devices, the existing resources for us to roll out this system. So the facilities already have the tablets and those are the resources that we are using for data capture at facility level. Then we have the lab request and the lab result stage. That request mainly is captured by the district health management team. While the lab results stage is mainly captured by the laboratory technicians from the different testing labs. Then we have different case investigation forms. Each disease has a case investigation form besides some diseases that are classified like for viral hemorrhagic fevers classified in Ebola and other diseases. Then we also have final outcome that closes the entire case. So this is the entire flow of the program. So this is the data flow of the case-based disease surveillance system. Everything starts from the health facility and the health facility notifies the case using a tablet that is the Android application. So then a notification is sent through SMS and email to the district health management team and they fill a lab request form based on what they find on ground when they visit the facility. Then depending on what assessment they've made and then they'll have to fill the lab request form as well as capture it in the system. At the district health management team they're able to use either the android application or the web application on the computer because they have access to both the tablet and a computer we'll have to. So they go ahead and also capture information on the case investigation based on the disease that is being reported. Then the lab that receives the sample also captures results into the system after analyzing the sample that has been collected. Though we still have some gaps on the results capture the lab is still resisting on capturing lab results directly into the system and currently we receive results through Excel files. Then the surveillance team will be the one to capture the lab results directly into the system from the Excel files. Then the national level analyzes the data that is sent from the different reporting levels for decision making at the emergency operation center as well as sharing information with other ministers of health departments and other ministries in the country. So as I mentioned that we use the existing case-based disease surveillance system to cover COVID-19 in February this year. So COVID-19 as because it is a respiratory illness it was categorized under the group of acute respiratory illnesses where we have the SARS, the MERS and influenza. So that's where we categorize COVID-19 and we use the WHO existing forms the investigation forms the contact tracing forms then that is what we configured into the system to meet the reporting needs for COVID-19 and initially it had questions of animal markets contacts with animals which eventually became irrelevant in Sierra Leone. So we went on revising the form eventually we had to capture in some other sections that are relevant in Sierra Leone aspect compared to the general WHO assessment. So we also added the contact tracing and quarantine monitoring tool which is still being used right now so we have two programs we have the NMC which is covering the information on the COVID-19 investigation then we also have the contact tracing which is a separate program and contact monitoring then we also have the port of entry passenger locator event we were using this for air travelers which we are going to expand to land travelers as well currently Sierra Leone is planning to open up land borders so we are going to use the same event program to capture data on on travelers through land currently we are using it for travelers through air also we are using it before at the beginning of the outbreak before the airport closed so it was very informative for us to tell where the cases were coming the confirmed cases we are traveling from and so on and so forth including quarantine monitoring so we also have a program on quarantine site monitoring and this is an integration from another application called com care which has been introduced by the directorate of science and technology um so um this program pulls data from the com care app into um the the tracker program in in dh is to which gives us information on the different quarantine sites um during this outbreak so this is a multiple data flows into um electronic space this is a surveillance that we have currently for COVID-19 um it all starts from uh from a suspected case which might be reported either through a health facility or through a toll free line um from the community which is free to everyone in the in Sierra Leone then um then it gets to the case investigation team and the case investigation team will have to to get on ground either the health facility all in the community and fill the case investigation form which was already configured in um in dh is to then the data team goes ahead and captures this information into the system um same applies to quarantine um before I move to quarantine the case investigation team um will have to as well line list um contacts of the case that has been identified then um the sample will be collected and sent to the lab so with that uh when lab results return as I mentioned earlier that we still have a gap with the laboratory team the laboratory team um does not capture the results directly into the system but they share the results in excel so the surveillance data team goes ahead and captures the results in the system on behalf of the laboratory technician so that is still a gap on our side um but we are still pushing on to um encourage the the lab team to capture the results directly into the system and also looking at the workload um during the outbreak the lab team is so strained and stretched um to the extent that they are unable to to capture data directly into into the system so again uh when it's when I go back to the floor we also have the quarantine team that has to keep the to pick the contact list from the system that was already captured from the uh by the data team so this is what they use to keep monitoring um the contacts that are in quarantine uh until they complete the 14 days so in addition there's also a small daily aggregated data from com care that I mentioned in the previous slide and that we can also use to assess how many um how many contacts are quarantine in the specific quarantine sites um then we also have the case management team the case management team updates data on uh on the admission information of the cases and also um outcome information of the cases deaths recoveries evacuations and any other um clinical best information from their end so we have um some key achievements and challenges that we we have um we're able to capture information on uh the confirmed probable suspected cases on time um because the issue district is able to capture this information on their end and national level um analyzes and is able to make decisions based on that data there are minor discrepancies between the numbers that we have in the system vis-a-vis what the the situation room um is reporting um manually there's also limited capacity to capture all the negative lab results so uh focus is given um on the positive results then negative um we compare the excel sheets shared by the lab and we're able to import them into the system we've configured key indicators the confirmed cases confirmed deaths confirmed recoveries and so on case facility rates interpret and we're able to respond to any data requests from um from partners and ministry of health as well in relation to to the outbreak um there also no address geodata geodata into the system because data is captured from the hard copies we have a gap of um of uh tablets so um users are not able to collect this information directly from the field so we use addresses that are captured um in the system for us to get the the geolocation and we're able to get the maps of where these cases um are coming from so we uh we also have information on the contacts and quarantine monitoring through the system and this is also done at district level so it is um it is accurate data from the district and we're able to get the transmission chains um from the system with the transmission chains we are also making use of uh of geodata with also the relationship um applications in dh is to to come up with the transmission chains um data integration with also case management for outcome and clinical data this um was um was this started the midway of the outbreak um it wasn't um something that was easy at the start to to incorporate case management data into the existing surveillance data um then we're able to to capture data using android and currently there's a rollout at the facility level that is taking place though we still have challenges with the sms channel um submission so we are relying on zero rating the instances um for for for submission email and sms notifications are also active at the different reporting levels and we've also deployed the system for COVID-19 in all the 16 districts so all that 16 districts are able to report um COVID-19 suspected probable and confirmed cases including contacts and quarantine um cases uh contacts from their different districts but the challenge we have is um some facilities have all devices that uh 4.4 4.4 KitKat which uh which we are facing challenges with um using the application then we've also had the mobile device management previously we are 80 percent of the devices we are configured however it was getting it was becoming costly for government and now we've resorted to using up lock which is uh which is which can't be accessed remotely so we still have that gap and we have visualizations with uh with data from the ports of entry with data from case based um surveillance contact tracing and line listing um the traveler quarantine follow-up and case management we are able to pull out information um such visualization the maps the tables the graphs um and transmission chains as well then we also have displays we are making use of uh the display application which was uh developed by his Uganda so we display on the screens at the emergency operation center including the the situation room of um of the response so they're able to see up to this information from time to time um some examples of data use include we've um we've used the reports the maps the charts um in the situation room to make um decisions we are using the same information to external portals so um external portals with um directorate of policy planning and innovation pulls data from the case based disease surveillance system um on um on the from the data of COVID-19 then we use that data to also determine the hotspot areas for surge for sentinel surveillance and monitor the transmission chains as well um data trans data dissemination also to other ministries for example ministries of gender for inclusion of gender um in the treatment centers and also um in quarantine and same applies to a directorate of nutrition to um to include um child food and diet to consideration of the treatment centers and uh for quarantine homes and facilities and also a psychosocial support um this data is also used to make public health decisions like like opening schools and um and so on so the next steps with the ACBDS rollout is we're looking at the transmission chain analysis using the relationship app supporting multiple tracker uh programs so as I mentioned earlier this one we are looking at uh using two applications DHS2 and GoData for comparison and uh we are also looking at including clinical data from case management currently um what we have is the out the outcome information that dates the recoveries and so on we're also looking at operational data from case management the bed capacity bed utilization and the admission rate and so on um then also the com care application which we've already started um the integration this is something that we are also continuing with so that we are able to have data from com care application flowing to um ACBDS then integrating the E-man first application which was developed by another by another agency into ACBDS so the main aim here is that we are focusing on having um this system be the center for COVID-19 data so any other um small applications we integrate them into uh we integrate them to ACBDS so that it becomes the central um the central data for COVID-19 then we are also looking at a possible inclusion of key logistics data like masks availability of PPE drugs and so on um we're also looking at uh tracking other conditions like maternal deaths and perineto perineto deaths um as independent programs within the system then also better analytical integration between ACBDS and EIDSR EIDSR is the weekly aggregate data which is already rolled out throughout the entire country in the 16 districts and they've been reporting for the past um one and a half years through that platform so we are looking at comparing the cases that are being reported um within the case-based disease surveillance um vis-a-vis what is reported on the weekly basis um and also um including the disease-related monthly aggregate data um thank you thank you Richard um I'll now give the word to Kalle and remember you can you can write your questions at the community practice uh if you do that it's also a possibility for us to get back to these questions uh later also after the presentations and the conference all right good afternoon everybody this is Kalle here talking from Oslo are you able to see my screen yes yes all right so I have only about five minutes so I will mostly talk about the more sort of strategic uh aspects of this development now at the beginning of the COVID pandemic what was interesting was that in almost all the webinars that I took part in everybody with experience particularly from the Ebola epidemic was stressing the need to use existing systems and that was a hard one experience during Ebola we saw how an epidemic or a pandemic with a high public profile ends up being a proliferation grenade or a fragmentation grenade with lots and lots of different actors and stakeholders jumping in and saying we have the solution for this we have the solution for that and the net result of it is that the uh the um both the public focus and the the management focus tend to be sort of diversified people are tracing up all kinds of avenues and very often the basic methodologies that are known to work in which the health sector is familiar with ends up actually suffering and not getting the support that they require uh I'm also going to talk a little bit about integration between IDSR which is integrated disease surveillance and and response and um and yeah some of the implications for DHIs too so the problem we have seen and I guess most of you are pretty familiar with the the the scene in the United States which is in a way extreme because the US have gone from a situation where they during Ebola actually had a very coordinated and concerted effort and I think played an absolutely vital role particularly the US Armed Forces but also CDC in assisting the three countries in West Africa to to beat down and and finally get rid of the the epidemic they've gone from that to a completely I would say almost chaotic situation where politicians where scientists where all kinds of conspiracy theories etc is constantly creating a scene of chaos and pulling also the population in all kinds of of directions and I want to go into details on that but you've seen similar things in a lot of other countries to a greater and lesser extent few countries Sierra Leone for instance Norway for instance considerably better I would say than average but even here you've seen some of the same things about you know new apps being thrown out as the solution to things and then after two or three or four weeks it turns out to not work you obviously have consultant companies you have all kinds of app developers for the best of of intentions you know pushing their apps and seeing it as a way of I would say making themselves known as a way of marketing and finally which we have seen to some extent in in Sierra Leone I'd say is that within ministries and within sections of ministries these pandemics ends up being also venue for sort of turf force different bodies are all paying lip service to full integration and full coordination etc but in practice they are actually doing their own things and and you know waving the flags Sierra Leone we've we've seen a number of new apps and whatever being pushed out and even if so far at least everybody is saying that well the data should go into the cbds but so far it hasn't and again it comes back to this this fact that during normal times you see surveillance particularly low income countries but also in wealthier countries is not seen as a very high priority it's seen as boring it's a lot of drudgery you know you're you're monitoring things on a on a day by day basis for years and years suddenly something happens so we've seen it in Europe for instance we're very very very few countries have still maintained their large the depots for instance of all PPEs and other kinds of equipment most of them found that to be not cost effective so they can rid of it then you have a pandemic and you are in a crisis mode everybody is screaming and fighting over the same resources so the little bit of the background was then that the cbds was based on the template that I was part of developing for South Africa which is used there for malaria at the moment based on the fact that normally a notifiable diseases will have one common registration form for all diseases and they would typically have one sort of process for dealing with laboratory data and getting laboratory results back but then it diversifies based on the disease although again we we decided in Sierra Leone that most of the acute respiratory infections are so similar that we can group them together and you can do that with some other diseases but typically you have a diversification and in tracker program terms that means that you have multiple case investigation stages basically one for each disease because what you are investigating will vary depending on on the disease and but it was the whole design has been that it's very easy to add new diseases it's easy to add new processes follow-ups like like follow-up visits and so forth because you can just add another stage you can add a few data elements etc so it was designed to actually grow over time because we know that you know since 2000 we've had SARS, MERS, Ebola, smet flu, a range of zoonotic diseases particularly but new diseases and we know experts estimate there's around 40 000 viruses in the animal world some of them will definitely jump to humans become zoonotic diseases so you know it just doesn't make any sense that with every new zoonotic disease we end up having to create a new set new databases new tools new everything it just don't make any sense and I think it's also often underestimated the strain this poses on the staff because again one of the things we've seen in Sierra Leone because we were we had just sort of piloted the ECBDS when the pandemic struck was that we had huge we've had huge problems weaning people off just using their own excel sheets we in sort of parallel to the ECBDS capturing all the data and providing it and increasingly taking over people have been running all kinds of excel sheets you know in in sort of parallel which we are only slowly then managing to to to pull in another future aspect which we know gonna gonna grow stronger is all the ideas around big data social data mining all kinds of variable devices etc etc right these torrento technology solutions will just increase but we also see that few of them live up to expectations at least for now right there's a lot of hype very little actual impact on we see now with the pandemic most countries ends up predominantly relying on contact tracing on following up on treating people efficiently etc and on traditional methods like social distances wearing masks etc right so these fancy new technical solutions generally don't have much impact on the epidemic it might change in the future but for now it's really really critical that we establish integrated systems that can be maintained that are not too fragmented you know so if you're if you're sitting with five or six or seven or eight databases is much more resource intensive than if you're sitting with one or two so we need to maintain this sorry may i ask you to wrap it up now leave some time for questions please yeah and and the the critical part here i think is in addition to maintaining an integrated ideas our system is also to start more actively combined surveillance data and normal service data which is traditionally monthly and aggregated but increasingly also might be tracker type of data and it's critical i think going forward to better uh shall we say design the health data so that it's possible it's it's more possible it's easier to combine that with social and economic data to assess impact of different interventions that you're doing in a pandemic situation and the implication for dhs2 that's my last slide here is that it fits as an integrated platform these key requirements but the current data capture is not optimal for ideas are we are working hard on trying to to to shall we say fix that with a new app trap more possibilities for line listing type of editing and a range of other things that makes it easier to use for these specific things we're still having some challenges with like the sms channel on android but again that's also being resolved and we are working on improving analytics and and line listing what i think we are missing is better tools for communicating this surveillance data with population so not only to other experts it's as i said integration with social economic models and it is generally to increase the awareness among decision makers and politicians that health and disease is equal to security and it needs the same time of long-term investments as as politicians general are very happy to put into you know military hardware and that kind of thing thank you okay thank you bridge it and color we have time we have five minutes it should quit a bit early to leave time for people to arrange the next session but i'm going to share my screen now and show um if you stop sharing your screen color there are some questions on the community practice feel free to add more there if i share my screen and um you should be able i hope now to see some questions and i believe we can start at the at the top and there's a question here i hope you see it bridge it from surva dal nielsen you mentioned resistance among staff to record the datum system can you say a bit more about that did you investigate the reasons for this thank you dalsen one is they they are resisting change they feel more comfortable using the excel sheets that they've uh that they've been using um despite several trainings that they go through they feel more comfortable um capturing data on the excel sheets but also connectivity um could also be a challenge um because we have labs um the district not in the capital city as well so um the fact that we don't have the sms channel working yet um data bundles becomes a challenge um that's why i mentioned that we are looking at zero rating the the the instances so that this can no longer be a challenge um but also they feel they feel um they are taking a lot of time um capturing data through the the the the the web best system as well as um they need to maintain their excel sheets so they're looking at also maintaining uh parallel systems um one other thing that they that were raising was um um confidentiality of patient information um which we we we we explain to them that this data is only accessed by people who have um access to it um who need to have access to it so as as we move on we are we are able to keep explaining and you know um convincing them to to accept that change from excel to um to the web best platform so so further that improvement um um compared to how we started if i can add something here also comment on some of the other questions let me say resistance to change is part of the reason we also saw when traveler started paying for um for lab tests that the the lab tended to prioritize that so there might be you know money might play a role here too it's often one of the reasons why different sections want their own systems because it's a way of getting additional resources but in all fairness uh last week when somebody was sort of questioning the labs uh why they were resistant to capturing data directly the response was quite swift and the person was invited to come to the lab dress up in full PPE gear spend a few hours in a lab that deals with live Ebola viruses and live COVID viruses and see how easy it is uh with with hands in gloves and whatever to actually use the current technology we have with the touchscreens and whatever so there is also an aspect of that I don't think most of our tools are very suitable for hands in three layers of gloves etc we also know if you have have gloves on how difficult it is to even you know deal with your own smartphone Eric is asking about the number of cases well we're talking 15 to 20,000 cases I know that I have so far you located about 12,000 between 12 and 13,000 cases so totally we're talking close to 20,000 obviously a lot less than some countries but also not not small numbers for a disease surveillance system and bram is asking about aggregated weekly report and case-based report data and for now that's definitely happening in parallel and the data is not partially it's used to identify gaps and whatever but it's also not directly relatable because the weekly aggregated data are suspects whereas in the case-based system we are obviously wanting to know which of the suspect cases are confirmed cases and this is the same for malaria systems you're normally focusing predominantly on confirmed cases but over time I think those of us who look some years ahead here we can see that for all diseases with reasonable numbers and that means up to two three four five thousand cases maximum per year it makes sense to move to case-based disease surveillance and then gradually phase out the the weekly aggregate stress becomes aggregates for the for the case-based data but for for some like malaria in Sierra Leone is also notifiable and they have you know two three million cases or more a year there is no way that that it doesn't make sense to have that as as for case-based disease surveillance because there is no capacity to investigate and no meaning really it has it doesn't really have any meaning to investigate other countries like South Africa which has 10 000 cases they that's all case-based and not no no aggregated data at all we're doing completely away with the aggregated data collection so this is a process and I would expect it to take two three four years thank you okay Kalle thank you I think we have to stop there but please keep adding questions if you have and I encourage Bridget and Kalle and others to to visit this page on the community of practice again to follow up on this discussion thank you to everyone who came to listen to this and the big special thanks to Bridget Magov and Kalle Hedberg for presenting and I'll see you in the next session okay thank you everyone


doing something yesterday and it failed so okay recording is started. Perfect. Hello everyone, welcome to the Jenkins Infrastructure Weekly online public meeting. We are the 22 of February 2022. Today is an is it anagram? You can take the date from left to right or right to left. Hey, nice. Okay, so today we have Mark, Stefan, Tim under with us. We're missing here, Stefan. I'm here. Nice. First announcement, so I saw notification about release. I don't know if it was finished for today's release, 2030-35. Tim or Mark, do you know or don't? No problem. I haven't seen anything, so I'm guessing. So I have seen that the releases are published to Artifactory. I have not checked that the jobs have completed successfully and I haven't done the checklist. So I don't know and I haven't done any investigation on a docker image or a docker tagging problem. It was reported by someone with grad to JDK 17. Okay, currently checking. We have an automated job that's rebuild the image, so quite and it's visible on Jenkins. I can check it separately, Damian. We don't need to take the time here. If the weekly has not completed, I'll chase it later after I get my problems resolved. Okay. That's the fix today. Okay. But at least on Jenkins IO, it's advertised that it's released, so we only have to do the last checks. Just to note, last weekly introduced a lot of GUI changes. By changes, I include new features and graphical bugs that should have been fixed this week. So thanks for anyone reporting these issues and fixing these issues. Do you have any other announcement? Tim, could you merge the changelog for 2.336 on jenkins.io? I think we already approved it. The docs office hours looked at it multiple times. Looks good. Yep, doing now. Thanks. Yeah, I see the changelog started for the next one two hours ago. Cool. I missed that one. I must have actually know I'm just way behind, I think. So as a reminder, the point for us as infrastructure team is also to ensure that the tooling that we provide to the release team is working and is stable over time. So that sounds like last week happened without any hiccups. Sounds like this week also with the automated. So for once we have two weekly straight without any error on the tooling part. So let's see next time we change something in the system. Just need some IRC notifications when the build starts and finishes. Correct. Good point. Good point. Good point. Okay, I don't have any other announcements. So I propose we start straight to it. So first of all, what did we do? A congrats survey for the work on digital ocean. We are now using digital ocean machines in productions. The Kubernetes cluster has been added to CI Jenkins IO transparently. There are some jobs that run or are currently running. We haven't checked strictly the metrics, but we could because it's reported on that as well. But we saw some activity. So these are about almost 30 jobs on the digital ocean cluster only. I don't know if I can share you a screenshot here. No. Don't worry. Add the screenshot afterwards on the meeting notes. Yeah. I publish it on IRC for now. So we will have to start checking after one week of full usage the costs that we consumed on the OKS, right? Because we have added auto-skilling. Yeah. Currently with 200 pools we have, there is a minimum amount of $350 per month and it can go up to $3,000. If the auto-skillet node goes up to 10 nodes. Okay. So we might have to be careful on that part and maybe reach out quite quickly on two digital ocean themselves. So yeah. For the costs to be checked after one week, after one week, how much did we consume on our credits? I also added this morning labels on these Kubernetes agents. So they have their name in it if we want to target a specific provider in our test jobs. So you can specify your CI, KB test, or Geo KS level. Okay. Thanks a lot for that. That's nice if you want to avoid that cluster or specify it for a specific test. Erwe, did you write an email to the developer mailing list? Okay. So that's to the list. Let's say cost, measure cost consumed. And also, do you mind updating the CI Genkinsia documentation? I will give you the link. The page where we list all the agents and labels available for developers to add the two new Kubernetes labels that you had it. So they know they can opt in or opt out if they start to see issue and there will be autonomous for continuing working if we don't respond immediately. Are they already in the sponsor page on Genkins.io? If not, maybe. So first the first the doc for developer. Yeah. Yeah. Yeah. Okay. And then, yes, we have digital ocean. We need to improve and nurture the sponsorship. We need to check if they, as you said, are they on the sponsor page that I don't know, Mark or Tim, do you know or even Erwe? I'm asking. I didn't check. I thought we had them added, but it's an easy check. I can take the action separately or others can check it. Okay. And there is a blog post to start on that topic on Genkins.io. I can't see them. I can't see digital ocean on the sponsorship page. Okay. So Erwe, do you mind driving also or just to do the pick on digital ocean or ask someone else directly? The goal is not for you to do all of them if you can't or don't want, it's just for you to drive and ensure that someone do this. Tim's right. We've got Datadog, Giscours, Fastly, the kind of entry for digital ocean. Yep. Okay. Nope. They should. So it's definitely in the to-do list, right? Yes. Cool. Are there any question, interrogation from things on the digital ocean? Cool. Next step, the other major topic particularly targeted to the infrastructure cost is the Azure AKS cluster. That's the top priority for now, at least from my point of view, because first price, second security. We now have a general reusable Terraform tooling that we shared on at least three different Terraform projects, digital ocean, Datadog and AWS. So the next step now is to re-boostrap the Azure Terraform project that used to be defined as code. And then it has been run last time before I joined the team. So at least one year and a half ago. Since then, the configuration has been changed only manually on Azure. So it has drifted from the config as code, even though everyone tried their best to keep it updated. Thanks, everyone for that effort. So my proposal, that's a proposal so you can say no, you can stop me. But my proposal is start from an empty repository again. We can always use Git to consult the order knowledge if we need to. And we start from scratch. And we'll start to add Terraform resources, so the new private cluster to be created, migrate everything then. And then we start either import existing resource or create new ones on that project with modern tooling, without interfering with the legacy states. Does it make sense? Is there anything that could cause problem? That sounds good. The main thing that's good for my point of view to get sort of will be the DNS, which would just be an import. The upcoming resources I see in the short term are the new private cluster, I guess, instead of the temp that we have right now, the DNS, reference to the CAA DNS recording, and the new MySQL database for writing. That's not priority, but that's a good exercise to get started with. That's the status on Azure. On the area of our dependencies to be kept up-to-date, it's like a never-ending topic, of course. Thanks, Stefan, for the effort on keeping all the HashiCorp dependencies updated. You drove that change. So Terraform, Terraform modules, Terraform provider, and Packer, all these HashiCorp tools are kept up-to-date, thanks to UpdateKli and Stefan's work and tracking. There is some work around Golang when we need Golang on some Terraform or OpenVPN repository, for instance. This one weren't updating since a few weeks or months. So Stefan is currently working on that part. It's not as easy as it is. So almost there. We plan to do some contribution to UpdateKli because either we are slowed down by, let's say, unwanted behavior, or we want to improve our ability to define an infrastructure resource on one repository, like with Terraform, example security group for AWS, and then get the real name of that resources tracked by Terraform and update it automatically on another repository, for instance, Jenkins in Jenkins Configure code, or update automatically documentation, like you add a new label on Jenkins, you want it to be automatically updated on readme.markdown, because these documentation are important for our users. So great job. Yeah, let's go ahead. Is there any question, things are clear or things I could have forgotten on dependency topic? Okay, continuing then. There has been a request from security team around AccountApp key cloak. So it's not the first time that this request is emitted to the Jenkins infra team. It has been for months, even years. That's the project of migrating account app features into key cloak installation. So account app is an application that helps a user to create and manage their account on our LDAP and centralized authentication for CI Jenkins.io, for Shira, all other items. Key cloak is the same, except that key cloak allows us to have a public front end and admin back ends for administration. It provides way more features. And it's not a homemade application like account app. So the risks are, let's say, a bit lower using that. And that project was never finished. There were some blockers and I will need help from the people who worked on that in the past. I understand that the work that Danielle did on the matrix 3.0 Jenkins plugin, remove one of the last blockers. I don't know if there were other blockers. I've listed here on pure infrastructure, the three main blockers or almost blockers, a step that we need to fix in order to fully switch out from account app and dump that project forever. Not that it was really useful, but key cloak is safer as for today and for the future. So I don't know if there are other elements that we did not track on that list, things that you would remind Mark or Tim or the others? No, it was just a matrix 4.0 thing and the group naming thing or something where you could create a user, you could possibly create a user with the same name as groups. The key cloak experiment is from a long time ago. Other tools have been studied like decks. Have we to go with key cloak, still go with key cloak? I don't know. Because there are some orders with some, I don't know. Decks wasn't providing air back. Decks was only about identities. But it was some time ago and maybe the situation has changed since, I don't know. Good question. We'll have to ask... And Olivier. So just a note on that. If we are going with another tool, there is a security assessment to be made, I think, as it might have been for key cloak already. I don't know. One thing though, we have clarified with Vadek the fact that the security of the infrastructure in itself is the responsibility of the infraofficer. It was a long running subject between Olivier and Daniel when there were respective security and infrastructure officers. I don't know if the board had time to validate that change proposal. I don't remember. For me it was okay and for Vadek. That means that the account key cloak sits in the middle of these two areas. So that's why we should ask them. But we might expect that it's also our responsibility to define that. So we have to ask them for what are the requirements from their point of view. So we need them to act as consultants and not as doers on that. Sounds good for everyone or are you agreeing, is agreeing? Yes, I agree. Thanks Mark. So I don't know if anyone is willing to work on that topic. Let's keep it. By default, I plan to keep it as infrastructure officers since it's my security brand new responsibility. But we need to finish the private Azure AKS thing because we need a private cluster to run a key cloak with it. Another small request that anyone could take. We have a Jenkins controller named CERT-CI managed by Puppet likes Trusted and CI.j. It's only partially managed, meaning we managed the Docker-based image and the virtual machine, but we don't manage the Jenkins configuration. That instance is dedicated to the security team. Badek has asked us on other channels if they could have Windows agents. So the goal is to add the same cloud configuration for GICAS can enable it on that instance. So they would have the same virtual machines, ephemeral agents as we have on other Puppet manage instance. So that's the requirement. They are okay to add Jenkins configuration as code to configure agents. So if they are okay for that, we just we only have to proceed. That will be managed adding Yara data for that specific machine that should be work with the same image and semi-config exception. I've listed what is needed. Two elements are as code. We might need to ask them to insert a credential because credentials are manually managed. So I don't know if anyone would want to try this one. Do you think I'm able to? Yes. So I would if you agree. Yes. Is there anyone to gain? So four. I agree for Stefan doing it. Yes. Plus one for me as well. That sounds great. Yeah. The only thing I'd say is generally your default route doesn't include that instance. You need to modify your config on the VPN or locally on your machine. Oh, yeah. Yeah. In order to get access to third CI, right? Yeah. I mean, you can just add your route to it locally or you can change the VPN to add your route to push your route to it. It's the same thing as third CI. Sorry. Sorry. Trusted. You need an SSH tunnel to reach it. No, you don't need an SSH tunnel. You just need a route added to it. Okay. Thanks. Yep. Let me write this down. I'm sure I remember where the VPN config is. Is it in the local VPN? Yes. Don't worry. We will double check this one. That's a good reminder in case because we would have forgotten otherwise. I think the runbook might have the definition, I'm sure. Yeah, at least once. Cool. I have access to third CI by default. So worst case, I can always compare other settings to mine so we can find what is missing. Yeah. You have access. I think you're one of the few. Okay. Thanks. So, Stefan. Thanks. That will be your next task. We don't have an help desk issue for this one because third CI is kind of, let's say, area that can go some city very quickly. So that's why we have to keep talking internally. I don't think that's sensitive. That should really have a ticket. Yes. If Vadek is okay with that, I'm all on help desk. So your first mission, Stefan, is ask Vadek. Convince. Convince. Yeah, it's just they are okay for us to speak it publicly on that meeting. So I agree with team that should be on help desk, but we never know. So better to ask first, better safe than sorry on that area. But yeah, you're a correct team. I agree with you that should be public. Easier to track and to delegate. Yeah. I think you might have added third CI to most people anyway. I think he syncs the conflict to be the same for all the files in December 2021 on the 30th of December. Yes. So it's pooped up light. Olivier was able to negotiate with Daniel to enable it again because they were required to run the updates. So it's updated at least for the core Docker image. So that's why it should be easy. I just mean I think you've made all the routing the same for everyone at the end of December. Okay. So that should be okay. I think you just applied your conflict to everyone. Okay. So I didn't hear your the beginning of your sentence maybe. There is also an ongoing issue from security since a few weeks around disabling anti-spam for the cert team. I've put the link that's the help desk. Just a reminder mark. So around Christmas I think after in January a few weeks ago you blocked thanks a lot an IP at the table level on the CI Jenkins virtual machine because there was spamming CI Jenkins. It was done at the Linux scanner. Can you remind us or does someone remember because I was out so I don't know what has been done after that. I've done nothing since then. It's embarrassing to admit that I've done nothing at all since then that IP is still blocked and we just need to take an action to probably by now unblock it and watch that it's not an issue and get rid of that single IP table hack that I put in. Assume you just restart IP tables and it will be gone unless you modified it on disk. Yeah. I guess that we have may already be gone and great even better. Yeah. IP table just does it while it's running by default. You have to modify files to get it to save. I love the manual blocker. I'm taking this one if it's okay for everyone. Yeah. That would be great. Forgive my not not taking a further action but yes that more than happy to have somebody else undo my damage. The other thing back when you mentioned spam the cert because it's just a similar thing is that you can see our dev mailing list is still going into spam even after the change that Mark made a few weeks back on the Google groups. Right? Yeah. Yeah. I see the same thing for Jenkins users and several others. I have to just monitor my spam folder. Yeah. I'm doing that too but it's annoying. I remember Henri who shared something with me. My email foo is close to zero. I have no idea of this spamming stuff works. So I will defer to someone who knows or I need to do get myself complicated. It's really complicated. Unless you're Google, right? Yeah, but it's from Google. Google to Google is missing. I don't think we would be able to do anything about it. Okay. We reconfigured. It's Kevin who showed us the send by the mailing list or send by the user. It's apparently configured to be sent by the mailing list. So about that I don't see what we can do about it. Do we have DKM or SPF records or for Google? The settings available is you send the email from the mailing list or from the user. It's about that. I am not sure there are more parameters. Do we have an help desk issue or with people complaining or let's say elements that I could use to indicate myself? There is, but it was because we thought it was fixed, but it's definitely not fixed. So can you share the link on IRC or in the meeting notes? So if we see that again, can you share knowledge and elements on that issue so anyone could take it or start doing it? It's not a big priority, but it's really annoying for most users facing that. So if we can, anyone with email knowledge could help. That could be great. It was fixed, but no technical elements. All the stuff I see on Google is about spam two Google groups, not the other way around. And an upcoming topic. So what you said about IRC notification. So let's say a next big thing once most priority has been done will be enabling custom IRC channel used for notification, maybe multiple depending on the topic. I still don't know. That's another part where I don't know anything, but Hervé and Tim, you seem to have worked on Jenkins Infra-notification IRC channel. I don't understand why, but I can't debug it since it's running untrusted. In the Jenkins Afra-puppet code, the battler is configured to post and to join the new channel I've created. I can't debug it. I can't log on the machine to see its log since it's running elsewhere. We discussed about moving Infra reports from trusted to Infra-CI. I think this one could be moved too. Okay, so Infra-CI. So let me finish on the notification topic. So just three elements. We have puppet notifications. So just to say earlier, we have to work on that. I propose we'll work both of us Hervé. It's because we have the puppet master virtual machine that controls the puppet stuff are applied and it receives message when a puppet agent apply its puppet manifest on a given VM. That machine is almost not managed as code. It's kind of hidden somewhere. This is also where I understand based on the recent feedbacks that that machine is responsible for connecting to IRC and sending messages. So that's where we would have to interact. Maybe moving the puppet bots from there. I don't know how it works. We will have to diagnose at least the both of us. But about notification, we also have Infra-CI and release CI notifications. So the notification from these two Jenkins instances. Infra-CI could be on the same channel as Puppet. But the goal will be when we have one of our terraform, Kubernetes management, or even Docker images, jobs or packer. When the main branch, the principal branch of this repository fails to build, we should receive the notification at least. I don't mind for pull requests that will be too noisy, but at least the main on the main repository. So we know it's failing and we have to fix it. As Stefan discovered earlier today, OpenVPN wasn't able to wasn't able to update its dependencies in a few weeks, for instance. It could be worse. It could be Kubernetes management. So, yeah, sorry? No, I was thinking we can also have a specific channel in our proprietary slack, but complementary to this IRC notification. Yeah, first IRC. And as Tim and Daniel suggested during the past days and weeks, we might want also Jenkins release notifications that will receive messages from release.ci to help, or maybe the Jenkins release channel, I don't know. But when a weekly release starts, it sends the first message and when it's finished, it sends another message at least. And same for LTS. So that will be part of the release process when a release starts and finish. I propose we start with these three ones and see how it helps us. And as you say, they're very important ones in fraud reports to be migrated out from Trusted to CI into FraCI. This involves using switching to GitHub app. I will add there is an help desk issue for that. The initial trigger was costs and availability of agents on Trusted CI. And also, there has been issues reported by Raoul, Aruborosa, one of our contributors, where at the credential GitHub use for in Fra report is lacking some permissions and some plugins, resulting on the maintainer, the correspondence between plugins and maintainer and plugins Jenkins.io not being correct on some cases. It's the GitHub user Jenkins admin which was used to retrieve this data. And since Jenkins admin user permission has been reduced to the minimum in December, I think these data are incomplete since then. Exactly. So the goal would be to switch to a GitHub application dedicated only for that. Team gave to Irvine High at least the GitHub app administration right on Jenkins CI so we can create and manage the application. We need an administrator right to install it or update it on production on Jenkins CI organization, but that will allow us to generate credentials only for that that are strictly scoped and that doesn't depend on a GitHub account. So really better in terms of security because any admin can change it, but still it's a GitHub app that can be pretty tight scope. That GitHub app could not read private repository content, for instance, only plugins. Okay so that's a lot. I took on myself to add online 92 an option to handle the new in case of spamming on the kernel to use fail to ban to automatically lock IPs if you want. You can erase that or not, but that's 90. That's interesting. Never used fail to ban, but yeah that's you already did. I heard it was easy to manage, right? Yeah, absolutely. You just have to. Sorry. Go ahead. I think I also mentioned CrowdSec, IP blacklist, crowdfunded, but I don't know how easy it could be implemented. I think it needs an agent. I know fail to ban, which is quite easy, but I don't know about the crowd. Fail to ban is independent on CrowdSec. He's collecting all IPs of fail to ban from. Oh, you're merging all the information on the on the. Yeah, and it's crowdfunded. So you you benefit from. Okay, please correct my misunderstanding. I thought that fail to ban was a tool that given a list of IP address to lower deny. It was blocking automatically connection at the system level. No, fail to ban is watching your log files and is taking an intensive action depending on what you tell him to do. For example, if an IP fail on three or four times to log using the SSH team on SSH, it will be locked out, but we can do whatever we want as a declaration for fail to ban to ban an IP. For example, if we have log of CI Jenkins IO of a spammer, meaning that the that IP tried to do something three, four, five time and we got that in the log, we can tell fail to ban. Okay, you have to ban that IP and it will first ban it for like one hour, second time for two, eight and years depending on how often it's coming up. Okay, so in the logs, if correctly configured by checking Apache logs, could it do the same as what Mark did manually at IP table level? Exactly. Okay, so I understand correctly. So fail to ban and close sharing. Yep, it's not sharing any information with others. Yeah, like, like, quote, say, quote, say, it's sharing ideas that are completely different bad behavior and are kicked right away. There is no sharing with fail to ban is just using the current log in the system. Yeah, and I think not sure how crowd-seq is easy to implement. It might need an agent and so on. Fail to ban should be more than enough. There is something around crowd sharing that we need to make sure is that the sharing of the IP is compliance with any code. I'm sure you can only allow inbund, so you retrieve the external IP and you don't give yours. Yeah, but that's kind of bad if you're using the system. I will not tell you that, you know. Okay, interesting, I didn't know, I knew fail to ban for SSH. It's installed on most of the cloud server and works. Yeah, that's because it's a default way of working, but you got tons of ways using it. Okay, so I will check for that one. Thanks. Welcome. I think that's all. Do you have other topics or things you want to add or want to clarify? If it's working, there is an update about mirror. Not updated, I'm not sure. It's not only Alibaba mirrors. Let me find the issue. Oh, that could be the whole mirror system not updating. That one might be important to check first then. Okay, thanks, I missed that one. Yes, it's the helpdesk issue. This one. Okay. Thanks a lot. So we got to check this one. Sorry for one old topic. The 2.336 release is not complete, at least in the sense that I can't see Linux installers yet. So I've got to do some more research. Nothing, no actions from others required yet, but when I look at, when I attempt to do an install of a weekly on my test machines, the weekly, as far as I can tell, is not visible yet. It's visible there and that's a good sign. So just needs more investigation. I'll do the research later. Okay, just to be sure, because if it's available here, that's weird because it means that the Linux repository should have been updated already on their index. So that might be something weird on the deployment process. Thanks. Not to state if you need help or additional tests. Thank you. Cool. I don't have any other topics. Do you have some?  Thank you. Rock your boat. Have a nice day. Thank you everyone.