I'll be short. I'll just introduce our three speakers for this session. We have 45 minutes, meaning about 10 minutes for each and some time for discussion. We'll start with Celestin Hatageka from the Harvard School of Public Health, giving a presentation on the impact of the outbreak of Ebola and the free-care policy from Eastern DRC. Then we have Karen Grepien, Associate Professor at the University of Hong Kong, giving a presentation on the early impact of COVID-19 outbreak, also from DRC, or based in DRC. And then the last presenter, Angela Aaba from Affinets, looking at routine immunization data on DHS-2 in the context of COVID-19 from Nigeria. So the topic of this session is, you know, looking at data and service data in DHS to look at the impact of Ebola and COVID. So basically using the data on other services to evaluate what the current and previous pandemics do to the health service delivery. So without talking more, I'll give the word to Celestin. I'll stop sharing my screen and Celestin can share his presentation. Thank you. Thank you, John. Let me go ahead and share my screen. Hello, everyone. It's a great pleasure to be here and be presenting our work on the impact of the COVID outbreak, sorry, of the Ebola outbreaks and the free health care policy on the health service utilization in the Eastern DRC. Before I elaborate further on our work, I would like to provide a brief background to help establish the context of our work. Ebola was discovered in 1976 in the northern part of the DRC in the Mongora province. And since then, there have been several outbreaks including the current outbreak, the ongoing outbreak in the equator, and the two other outbreaks that took place in the 2018 one in the equator, and another one in the Eastern DRC. The outbreak in the Eastern DRC affected the free provinces. Those are Ituri, North Kivo, and South Kivo. Overall, South Kivo had a small outbreak which affected one health zone, whereas the outbreaks in Ituri and North Kivo were very large, and they were actually considered as the fourth largest outbreak that the country has experienced since the discovery of Ebola. So that's why we focus on this to understand the impact of the outbreaks and the policies implemented to mitigate its effect. So we know from the evidence in Western Africa during the 2014 outbreaks that Ebola outbreaks resulted in a significant drop in the service utilization. To mitigate this effect in Congo, the DRC government and the Arangüedizi partners introduced a free health care policy in some regions affected by Ebola. However, the regions did not benefit. So this provided us an opportunity to study the impact of the Ebola, as well as the free health care policy and service utilization at the same time. We leveraged data from DHIS to enable health information, health management information system in the DRC. Specifically, we use the data from health centers as we focus on looking on the impact on primary care services that are provided primarily by health centers. These services included total clinical visits, we looked at common infectious diseases including diarrhea and pneumonia. We also examined the impact of the outbreak and free health care policy using maternal health services including antenatal care and the delivery care. We looked at the impact of vaccinations. Given the HMIS data, the time series data available monthly, so we were able to design a strong study using interrupted time series with a control group. The intervention group on one hand has the region that had Ebola and free health care policy and then the regions that were not affected by Ebola and had no free health care policy were used as our control group. We also on the other hand looked at the effect of Ebola and the region with Ebola without health care policy or still using the region without Ebola and free health care policies as our control group. Although we use a control interrupted time series, a single interrupted time series can also be used when the control is not available as you will see in the following presentation by Professor Karen Grepin. When you're using these methods, the control interrupted time series, you're comparing changes over time before and after the intervention in the exposes versus the control group. We use specifically segmented regression approach to estimate the change following the outbreak and the free health care policy. What did we find? As you can see on these monthly time series of clinical visits, we found that immediately following the introduction of free health care policy in the health zone affected by Ebola, the clinical visits are more than doubled. This gain, as you can see, remained throughout the outbreaks relative to our control group, which is the orange time series here. However, when you look at the region with the Ebola without free health care policy, there was no immediate increase in the service use from the outbreak. However, service utilization increased over time at a rate of about 3% per month. This could be explained by increasing the investment in those areas to help fight the outbreak. We examine an effect on a common infectious disease, specifically the diarrhea diagnosis, as you can see from this chart, the introduction of free health care policy did increase in the diarrhea diagnosis, the rate more than doubled, and the gain, more or less, decreased over time at a rate of about 6% per month, relative to what you would have expected without the Ebola and free health care policy. But if you look on the region without free health care policy with Ebola, there was no significant changes. So looking at the range rates of pneumonia diagnosis, you can see, similar to what we observed in the diarrhea diagnosis, that the free health care policy deserted a significant increase in the service utilization reflected in the pneumonia diagnosis here, and the gain really remained during the outbreaks. However, in the region without free health care policy, but with Ebola, there was no changes observed relative to the control group. So when we examined the effect on health service utilization, specifically looking on antenatal care first visit, you can see there was a modest increase in the service utilization as expressed as an antenatal care visit. So about 17% immediately flowing the introduction of free health care policy in the Ebola region. Ebola health zones are relative to the control group, but there was no significant changes in the regions with Ebola, but without free health care policy. When we looked at facility delivery rates over time, we observed that when the free health care policy was introduced, as you can see here in August 2018 in the Ebola affected region, in the month following the introduction, the facility delivery rate increased by about half immediately, and then it continued to increase at a rate of about four per set per month when the free health care policy was still there. However, when the policy was discontinued in early 2019, the facility delivery rate immediately dropped by about 40%, and then it continued to drop at a rate of about 16% per month subsequently compared to what you would have expected without the drop in the service, the continuation of the free health care policy. This really showed how the free health care policy was effectively increasing the use of facility delivery. When we examined the vaccination rate, we observed there was no significant changes across regions with free health care policy or those without free health care policy compared to our control group. In short, our analysis really showed that free health care policy can be an effective tool to help mitigate the impact of infectious objects such as Ebola, and then we showed that it was associated with increasing services in Ebola affected regions, and its effect was largely sustained during the period the policy was still there. We were able to do this analysis thanks to a variability of DHIS to enable the routine health facility information data. So this data can really be helpful to examine the impact of outbreaks such as this Ebola, but also to really study the impact of policies implemented to mitigate the effects of these outbreaks in resource-limited settings. So I would like to acknowledge IDRC, which funded this research through a grant to Professor Karen Grape, and I would also like to acknowledge Bruce Quayer that uses his platform to provide us access to data using this analysis. Lastly, special thanks goes to our amazing study team here. Thank you. Thank you for joining me today on my presentation. Thank you very much, Celestine. Then I can ask Karen to share her screen. You have to share first, I think, Celestine. And while Karen is setting up again, I'll remind you that you can post your questions on the COP link provided in the sketch page for this session. And then we have some time at the end of presentations to respond to them. So Karen, please, you can start. Hi, everyone. I hope you can hear me. I turned on my video. I'm not sure if that means you can see me or you can just see my slides. But thanks so much for coming to the session tonight. My job is actually quite easy because this work actually builds quite a bit on the work that Celestine just presented. So as a result, the hard work is done and I'm going to mostly focus in on some of the results. So we're looking at a very similar kind of question and a similar kind of context, although now we've switched from Ebola to COVID. As Celestine has mentioned, there have now been three separate outbreaks of Ebola since 2018 in the DRC. And now, of course, they're dealing with COVID as well. So in early 2018, when the first outbreak, so this would be the first equity outbreak came about, we set up a research project to basically understand what we might call the sort of secondary health effects or quote unquote, the collateral damage of COVID. And the reason for this is that in the West African outbreak of Ebola, in the sort of post-mortem that was done after the outbreak had reached its peak, it was very clear that there had been very substantial declines in the use of health services in the very heavily affected countries that had Ebola. And this was across the board. It was everything from maternal and child health to immunizations. And the belief is that perhaps as many people died or had significantly more morbidity because of this lack of use of health services. And there's lots of reasons why that might have happened. This could be happening both on the demand side and that people, for example, are nervous or scared about presenting at health facilities when there's cases of Ebola. Or it could be on the supply side and that the health system itself was greatly undermined. There was a lot of issues around human resources. And so for a variety of reasons, we should expect there to be pretty significant declines in utilization. So when COVID happened, the fears emerged as well that this would happen again. And because of the work that we had been doing since 2018 in the DRC, it was quite easy to then sort of pivot what we were doing to say, great, now what's the impact of COVID on the use of health services in the DRC? Modeling studies that have come out since earlier this year have really predicted quite dire... Quite dire sort of consequences of what's been happening. I'm not sure, does everyone hear the interpreter? I'm hearing it. I'm hearing it too. Just a second, we need to adjust the settings. My apologies. That's cool though. I don't think I've ever heard anything I've ever said in Russian before, so that's quite neat. Anyways, I'll continue. So the pandemic in the DRC has obviously been quite unique. So the first cases of COVID appeared in the country sometime in early March. The initial detective cases were amongst travelers who were coming back predominantly from France and parts of Europe. They quickly mobilized to try to mitigate the impact of the pandemic on the country. And so like many countries around the world, public health measures aimed at trying to contain the virus. So for example, the closer of schools and bars and at the end of mass gatherings, they also quite severely shut down the amount of flights going in and out of the country as well as within the country. Big country and flights are an important way of getting around the country. And then because a lot of the early cases were really heavily concentrated in Kinshasa, the capital city, there was a lot of additional focus in terms of trying to contain the outbreak in that city. So for example, we see, actually travel in and out of Kinshasa itself being restricted around the end of March. And then the one neighborhood, which had one of the most of the cases, at least at the time was sort of appeared to be having a lot of the cases, was basically put into very strict lockdown. And this neighborhood is called Gombe. They are effectively in lockdown through all of April through the end of June. So this is a map of Kinshasa. It is one of the largest cities in the world. It's one of the most densely populated cities in the world. There's 12 million people living in this city. You can see here on the map, Gombe is one neighborhood. It is actually one of the more affluent and centrally located neighborhoods in the city. And it was where the outbreak was heavily concentrated. In the DRC, the health system is organized into what are called health zones. Each health zone would have a certain number of health facilities, including usually some sort of a hospital and a series of health centers. There's 35 that we use in our analysis in the Kinshasa area. And because Kinshasa basically was affected differently from the rest of Kinshasa, we've thought to also try to see if there was differential impacts of the outbreak in Gombe itself. So in terms of data for our research project, we use the same data that Celestone just described. So this would be the monthly DHIS-2 data from the national HMIS system. We use the data from January through June of 2020 for this analysis. We had a number of choices in terms of which kinds of indicators we wanted to use. Obviously, there's a lot available in the DHIS-2. We try to focus on indicators for which represent a vast majority of services available and also had high reporting rates. But we also try to get a comprehensive picture of what was happening within the health sector. So some that perhaps didn't have the best reporting rates, but we wanted to be able to target, for example, the non-communicable diseases as well. So a similar kind of approach to what we did in the Eastern DRC. We use interpretive time series. Here we didn't have a control group because everyone is, every area was effectively affected by COVID. And so we're really comparing districts to themselves in terms of being their own basic control group. This approach is great in that it also allows us to control for things like seasonality. We know there's a lot of seasonality in things like malaria. But also, for example, and this is a common problem I think in many countries that are just starting to implement DHIS-2. There's increases in reporting rates and increases in data quality happening in these areas over time. And so if we think that's happening in a reasonably constant way, we can also try to control for some of that. Another challenge we had in Kinshasa is there was actually quite a localized outbreak of pneumonia in December and February of this year, basically. And so we wanted to account for that as well. So what do we find? So first off, if we look at total clinic visits and look at this sort of before and after COVID, again, it may not seem obvious because you have to account for all the seasonality and account for these different changes that are happening in that spike that you see there related to pneumonia. But once we do that, we do actually detect a decline in total use of health services, which is perhaps not terribly surprising. It was on the order of about 20%. What's also great about the DHIS-2 data is it allows us to dig deeper. And so that's effectively what we did. And so if we disaggregate the areas in Kinshasa, if we look at Gombe, relative to the other districts, here we see there was a very dramatic decline in use of health services, specifically in the Gombe district. In fact, much of the decline that happened overall in Kinshasa is really what happened in Gombe. And so it's important to be able to disentangle those things. And if we look by facility type, we see that most of the changes actually happened in hospitals. And this would be perhaps consistent with the story of people being, you know, scared to go to hospitals because that's where they know a lot of COVID patients are being treated. We can also look at different types of health conditions and we did that. So for example, we looked at diarrhea. We also looked at malaria. Again, we see these important declines and most of them being targeted focused mainly in the Gombe area. Again, the area that was most heavily affected by the lockdown measures. When we looked at the NCDs, we had data on diabetes and hypertension. Here we saw very dramatic declines in the use of health services in, again, in the Gombe area, not so much in the rest of the area. So just to summarize, we measured a basically a 20% drop in all clinic visits immediately following the onset of the outbreak or the pandemic. And really this is really heavily focused in on the Gombe area. So an 80% decline in total visits there, and only 15% of everyone else. Rates of malaria, again, they vary by disease type and by area, but on the order of 90 to 80% for some of these in the Gombe region. Drops of non-communicable diseases, which is quite interesting. Actually, something that doesn't typically get measured in a lot of, for example, household surveys. We saw really dramatic drops and that's a really interesting finding. Perhaps people are pushing off those types of health services because they're seen as sort of less urgent. But overall, I think that the main finding from this work is that it's DHIs to data on these platforms and provide basically data to generate these types of results in almost real time. Obviously, there is delays in reporting and delays in us getting around to doing the analysis, but these data are roughly available and the work can be done. So again, just to thank the research team that has been contributing to this project over the last couple of months, as well as the Ministry of Public Health, who has been very supportive UNICEF and Blue Square as well. And again, this research is funded by IDRC in Canada from an actual old Ebola grant that we were able to pivot towards COVID. I put my email address up there if anyone has any questions about doing this kind of research in your own context, your own settings. We'd love to talk to you. So please feel free to reach out. That makes sense. And I'll stop there. Thank you very much, Karen. That's very interesting. So we have the next speaker, Angela Abba. And then we will have time for questions at the end. Okay, we see your slides now, Angela. Thank you. Good afternoon, everyone. I'm Angela Abba. I work with Affidavit Nigeria. And I'm making this presentation on behalf of other co-authors. So this presentation basically is looking at the humanization data reporting on HRS2 in the context of COVID-19 in Lagos State, Nigeria. All right, so this will be the outline of the presentation. We look at the introduction. We look at methods of what we are finding, the results, what are the key messages of management and diagnosis. By way of introduction, we'd just like to reiterate that HRS2 is the only government that recognize electronic reporting platform for health data in Nigeria. So it means that the humanization data is also reported on the HRS2 system. All states within Nigeria actually report their health data including the humanization on the HRS2 system, inclusive of the legal states. So in February 27, 2020, the first COVID case was actually recorded in Nigeria and in Lagos State. And since then, Lagos State has remained the epicenter of the disease in Nigeria, it's bearing about 3% of the body of the disease in Nigeria, even as a substitute. So the emergence of COVID-19 in Lagos State and Nigeria generally posed challenges to humanization services, beginning from care givers not aware that routine services were provided at the health facility and also people scared of going to the health facility for fear of contracting COVID-19. So therefore this depth review was basically just going to see what reporting was like within the context of COVID-19 and especially for legal states, it was the epicenter of the disease. So basically, we looked at four indicators that are monitored in country through humanization in the country. Before the cases are one, we looked at reporting. Basically we have four data sets in Nigeria that we use for reporting humanization data. We looked at the four of them, what was reporting like. We looked at sessions conducted for immunization. We looked at the next session and there's an active session. So we looked at both types of sessions. We also looked at coverage for some selected antigens and then we looked at supported supervision. So for this particular depth review, we did six months with prospective data analysis. We looked at period between January to June 2019, versus the same period January to June in 2020 and we did trend analysis. The map you're looking at is showing location of legal states in Nigeria. So these were some of the findings that we have following the depth review that we conducted. Number one, like I said, we looked at data sets reporting and we have four data sets in Nigeria. So this is showing the first data set, which is the EDGMI S2013 data set. We did observe that across all the periods, we had higher reports in 2019 compared to the same periods in 2019, 2020. However, we now see a different trend between May and June 2020. Reports from the EDGMI S2013 and we could see that between May and June 2020, despite COVID-19, reporting was much better compared to 2019. So this graph, this chart is showing us the second data set, which is the vaccine utilization data set. And this data set particularly reports issues around vaccines collected and vaccine usage. Again, we compared January to June 2019 against January to June 2020. And we could see from the trend that unlike the previous data set that we had looked at, reporting was much higher in 2019 compared to the same period in 2020. And we did notice that the period of March had a huge decline in 2020 compared to the same period in March 2019. And of course this could be due to lockdown of all the states during the COVID-19 period. Again, we have two other data sets. So we have a micro plan where the sessions that are planned and sessions are conducted, actually reported. And then we have the EDGMI S2013, which was an agenda document introduced after the routine immunization module was introduced to the CHIS set. So for the micro plan and NHMI S2013, we observed similar trends. Of course, reporting was much higher in 2019 compared to the same period in 2020. And just like the south of the vaccine utilization, we also noticed that the differences were highest in March 2020. And like I said, that was because the lockdown was actually, the directive for lockdown in the states was given in March 2020. Moving over to the next indicator, the next indicator that we looked at was coverage for selected antigens. And the antigens that we selected were basically ECG, then the three, ITD, measles, and yellow fever. So across all the antigens, we did notice that coverage was higher in 2019 compared to the same period in 2020. And looking at each of these antigens, you will see that the difference was more in April across most of the antigens. Of course, March, April were more of a lockdown period to expect that these people were not surprised about them, some of these findings that we saw. So across all the road periods for each of these antigens, the coverage was much more higher in 2019 compared to the same period in 2020. Number three indicator that we looked at was the sessions that we conducted. And like I said, for routine immunization, the two types of sessions physically, we have a fixed section and then we have the active sessions. So we looked at a number of sections that we conducted in 2019 and then in 2020. And of course, again, we observed that there were more sessions conducted in 2019 compared to 2020 for all the fixed and active sessions. So we could see looking at the active sessions by the right-hand side, we could see that there was really a decline in sessions conducted, especially for the most of people. Because at some point, the conduct of active sessions were discouraged in March, which was a curtain of the spread of COVID-19. We noticed that there was a 5% and 20% difference for fixed sessions and active sessions that were conducted within at least two years between 2019. However, we did notice that in June, sessions had picked up and we will see that we are having more sessions in 2019 now compared to June 2020 for both fixed and active sessions. And then finally, the last indicator that we looked at was the conduct of supervision. This is too usually. I encourage you country from the higher level to a lower level. And then we looked at this is that we conducted between 2019 and 2020. And of course, we did observe that there were more visits in 2019 compared to this. Overall, it was a 10% difference between on the set of visits that were conducted. So, in conclusion, we're both on the DHISP platform was observed in the lower end of this meeting compared to this year in 2019 in legal states. Of course, this could have been due to the COVID-19 pandemic contributing that there were restrictions on women, contributing that the states also would get sent out of the disease. However, we do acknowledge that other factors including funding and infrastructure that could have also contributed to some of the outcomes that we saw on the platform. So we want to acknowledge the current organizations in the U.S. and this is for June and after that, thank you very much. Thank you, Angela. And thank you also to Karen and Celestine. We have 10 minutes, more or less, for questions and I see there are at least one question on the community practice. I'll share the screen with you all. So we have, can you see this? I hope so. We have one question here from Kelly Hedberg. I guess that goes to all presenters now. The tools and models you're using. How can they be implemented in something like DHS2 or how can we as a community also make sure that these results and methods find a way back to managers and other users of the system? This is Karen. I can try to take that one. So in the DRC, the technical secretariat that is responding to GoFake has, you know, they have these dashboards that they're using to track and monitor. One of the things that we've been trying to do is to the extent possible try to move some of the analysis that we've been able to do onto that platform so that it might be able to feed in in real time to some of these dashboards. What we're doing is not particularly complicated, right? So it's relatively algorithmic and it could actually be integrated to some extent into some of these platforms.  You know, with Celestin's first presentations, we actually ended up spending a lot of time studying a policy change that happened. And as in the second case, we're looking really at how the outbreak is spreading across the country. So it would require some manual kind of looking through and finding out, you know, what's what's interesting to look at. For the most part, there's there's nothing that we've done that cannot be integrated into some sort of a platform that feeds directly off of the DHIs to platform. Okay. Would you like to add something Celestin or Angela? It's not, I have also a question. And it was mentioned at least by Karen about data quality being an issue. And can you say something about how you dealt with data quality or whether, you know, what were the data quality issues here without going into too much detail on how you solve this or something. But how do you deal with the data quality issues for this kind of research? I can jump in to try to answer this one. So the first one, I guess, Professor Karen has said most of the things that we've been doing in Congo, and then how the Ministry of Health and HMIS are also being involved in that. So in terms of assessing data quality, so we've tried various approaches to try to select indicators that are of a good quality as a current discussion or presentation. So one of the approaches we use first is to select to select indicators that are being reported regularly over time and it has a higher reporting rate for those specific indicators and we then consider them. And then we also look at the facilities. So there are some of the facilities that do not report consistently, because one of the things we wanted to do is not including what I would call body facilities, facilities that do not report on particular indicators. So we do consider indicators that have a very good reporting and then we also in for those indicators, facilities that do not report consistently, we drop them. So they were missing data is very common in this time city data, which, you know, range, which is has a, you know, have a period of two to three years, and there is an outbreak coming the reporting rate may drop or there are a lot of missing data. So we use the various approaches to also a roll some inclusion of indicators with the miss with the missing data, but using as acceptable approach to sort of account that in our analysis to have estimated that are not biased. I think that's about what we did on a indicators. Oh, I forgot one point. It's on the outliers. They also, we also consider the outliers. So our eyes were common in many of the indicators we use. So one of our approach we realized was to consider indicators. We are out where they are when the outliers to to to consider those outliers as a missing and then you treat them as a missing value. So the data quality has been improving. So however, in some context, you realize that there are a lot of missing outliers and missing. So if they were more consecutive missing data or outliers and then those indicators were excluded from our analysis. Back to you. Yeah. Thank you. Anyone else wants to add to that. All right, thank you very much. So I'm just to say that the response to the question. So, number one is that based on the analysis that we have gone so far, of course, we could see that there's so many measures that have been put in by the government to ensure that. And we could see that there are really senses have picked up, especially between May and June of 2020 and we're going to see much more better performance in this period compared to what we had in in in 2019. There are a lot of measures that have been put in place, communication, risk of rendition is ongoing and people have realized that services have been provided and services have picked up. And of course, during the analysis, yes, we're going to do this analysis here. We just looked at the first, first, the first six months of the year. We're going to do this eventually across the four months in the year just to see what happened in 2019 and 2020 and it really will be impacted by the nation organization services in the country and especially in the university. Thank you very much. Okay, so that was a good reply to call us the question here. And we still have time for for the last question by Anne Griggs. It is specifically about DRC and availability of data. So it's also a bit about data quality, perhaps. Yeah, maybe, maybe I can take this one. So, as I mentioned in the talk, we initially started this project in Ecuador province, which is, you know, one of the more rural and hard to reach parts of the country. And so we had a lot of concerns that the data wouldn't necessarily be of great quality and widely available. It turns out the data are reasonably good. And in fact, when we moved into the Eastern DRC, we also found somewhere levels of reasonable completeness and availability. In the DRC, you know, there has been a result space, format space financing program. And so a lot of reimbursement and stuff that is being run through that and therefore the DHS2 system plays a role in that process. And so for the most part, the data quality were pretty good. In fact, the data are probably worse in Kinshasa than they are elsewhere in the other areas that we've considered. So definitely is variation across the country that we've been able to find data of sufficient quality to do this analysis in all of these different contexts. Thank you, Karen. Okay, I think we are approaching the end of this session to give you some time to have a short bio break before the last January. So thank you to all participants and a special thanks to our three speakers, Celestin, Karen and Angela. And please you can follow up also with a bit more questions and discussion on the community practice page. Thank you very much.


Welcome. This is Jenkins documentation office hours. It's the 23rd of June 2023 Asia time topics on my list Google summer of code open pull requests of interest and a possible discussion and review of Java 11 to Java 17 changes in the documentation. That one I call a possible because I'm a little weary tonight so we're going to limit ourselves to probably not more than 20. 20 minutes. Any other topics you'd like to be sure we put on the agenda. Looks good to me. Alright, so I should tell us about how things are going on the Docker Docker quick start. So things are going a little slow this week but we have created a working example with jobs to it already has one job you can let me share a link. Great. Okay, so are you okay if we look at the link together or at the at the example. Yes, yes. This one I shared with him. Great. Okay, so let's. Here's what. All right. Experiment Docker. Okay. Second example. And so if I clone your repository. Are you okay if I just build it myself locally. Yes. Great. All right, let's do it. There's a key gen script before. Right there, you can run it to generate your own keys so that for connecting and agent Jenkins agent with controller. It generates new SSH keys every time to be secure. Okay, just a moment let's. Okay, whoops, I would help if I give the correct clone command. There we go. All right, so here is my.  And you say experimental Docker compose files and then oh too. This one, this branch doesn't have the SSH keys. Okay. Is in different branch. All right, so which branch is there a particular branch you recommend I be on one of the pull request branches or. Yes, this one first one new feature. Okay, so how about 42 feature. Yes. All right, good. Okay, so here we are. First render a script then. Compose up with deep. Okay, and so it creates. Oh good okay it does it locally so it's not not putting them into my into my set of keys. Okay, good. And then. Docker compose up. And I think that's is that enough right there just Docker compose up. You can add the option to the terminal later. Sorry, the minus the option for what. At three of the terminal after the containers are running. I'm sorry, I'm not. So Docker compose up minus. You're using the previous Docker compose here to add slash is between Docker and compose that. Okay, Docker dash compose. Yes. Like that. Yes. Okay. All right, so it's building. Downloading plugins. It uses a custom Docker file right now to build the image. Good. Oh, that's an interesting choice of upgrade state. That's fun. Yeah, that's we do plan to move afterwards. It's because of faster to make things faster go faster right now. Okay. So I've currently got something listening on port 8080. Let me do a quick check to see if I can interrupt it. I was trying to duplicate a bug. Just a moment. You got you got an error because you're. Yeah. And that's no shock. I knew I had, I knew I had a colliding service on it. So that's not a problem. I just wanted to see if my, my tests. Okay. My tests have passed far enough that the bug I was trying to duplicate can't be duplicated. Good. So I will just go interrupt that. Okay. Now let's do that compose up again. Okay. So now if I look at a Docker PS. There is an SSH SSH agent running separately and a Jenkins controller. Okay. And did it export the port 8080? Whoops. Wrong one. Home. Testing dash. It did. Very nice. Okay. Admin admin. Okay. All right. And here I am with an agent. And the agent has a. Nice job is not configured in this branch. It's in separate branch. I think this one was created earlier. And that one was merged. So job is not here right now. But it's added in the main branch. Excellent. Okay. I know I have to, I have to do some of my usual experiments. So let's see what kind of agent we've actually got there. And then updates. Let's update all the plugins to latest releases. And see if the updates persist. Across a restart. Okay. So it's restarting. I'd click the restart and will it keep running. Sure.  What was it on restart on failure? Yes. Thank you. Okay. So restart on failure is configured. Very good. Okay. So patients. Apparently I need a faster computer. Okay. Good. All right. And plugins are up to date. And Jenkins URL is empty, but I am going to configure it because. I know what it should be. And it got the correct value. Oh, very good. Oh my, very impressive. Okay. So now how did it, that's, that's, it's inside a container. And yet it got the correct address. Very nice. That's actually in my best thing. It was not getting better. I don't know how hard it is. Okay. Well, now I am running somewhere other than local host. Right. So I connected my web browser. To a fully qualified domain name. Maybe you connect your web browser to local host. Yes. Yes. That's okay. And that would do it then because. What Jenkins does is it uses the address in the web browser. Somehow to, to populate that field. Nice. Okay. Well, we do plan to use that update wizard. When we. Upload it and mainly so it will. John Mark mentioned that it will solve that problem. Very good. I do have an open issue right now regarding that warning that you are. Okay. And, and, okay. And I see it. Here's the indicator. All right. So this agent that you've got configured, there is running Debian Debian, the Debian Debian operating system inside the, or the Debian utilities inside that. Inside that agent. And if I look at the, the root content at the root machine. Let's see, how do I change it? Let's give it. Oh, it's got the same labels. Good. It's also running Debian as expected. Very nice. Okay. Good. They didn't have a job. You can run a job example job to test it. Right. So if I create. Hello. As a pipeline job and I go click. And say, let's do this. Hello world. Boom. Build now. And if you got declarative pipeline installed, that should work. You did very good. Nicely done Ashutosh. Very nicely done. Impressive. Thank you. Nice. Okay. So now what changed here? Oh, the SSH key changed. Okay. That makes sense. So what and your record? Oh, good. And you chose the right SSH key form. Well done. Thank you very, very much. Yes, we did have an error with RSA. It was not working that. So we shifted to. Well, and, and I prefer ED 25519 because there are at least some voices in the security community who say it's slightly more secure. And I like it because it's much less text. That is my kind of public key, tiny little thing. And so after this, we are working on the tutorials that uses Docker, the previous Docker commands. There are four, I think we have open issues regarding them and that and right now the mid mid evaluation is coming. So on 6 July. So this week, I think I'll be working towards that. Very good. Congratulations. So the ones that that that gives me hope that as you're making progress towards the ones that I was dreaming of were these right, the build a Java app. And so you're aware of these, this, this terrible, awful, awful thing here. Yes. These three and there is one with pipeline one also. Ah, okay. So there's, there's an additional one that I'd missed. That's. It's, is it this one? No, not Jenkins file under. It's on pipeline section. Oh, okay. All right. So it's probably this one end to end. Good. All right. That, that is marvelous. Okay. So, so just as a check, all right. And you very nice. You didn't include blue ocean in this particular image. Cool. But, but we also don't have the pipeline graph graph. Is that right? We don't have this one. This one doesn't have the tutorials one. We are working on tutorials one that. And which will include the pipeline graph. Okay. Instead of please. Oh, good. So you're going to go for pipeline graph. You instead of blue ocean in the tutorial. Yes, that's what. But we haven't decided it fully, but I was thinking. I certainly, I certainly like that. I think this is a good excuse to consider a break. To consider a break from the past, right? Because why not use the new thing. Instead of carrying along all the baggage of the old thing. So graph you pipeline graph you is. There we go. This one. That's exceptional. Anything else you'd like to show us. There was one example of job, but we already did a run job. So, okay. That's right. Now, and tell me about the storage. How does just from my own education. Does this use volumes? Yes. Right now. And so it doesn't use named volumes or unnamed volumes. Right now it does use name. Okay. So this is probably the name volume. There's zero two. Okay. So will that volume then persist. Even after I even after I stop the, the Docker container. Yes. If you, if you do want to delete the volume to you'll have to use Docker compose down slash less volume. Okay. So if I do, for example, if I do a Docker compose down now. It will preserve it. The volume will survive.  Let's see. I bet that is. Okay. These are the containers. Containers network and. Ah, okay. So it really when I did Docker compose down, it actually removed the network as well. But the volume is still there. Okay. And then when I do a Docker compose up. I'll remind myself Docker dash compose. And now if we wait patiently for this to start. And the job will still be there. It won't have been lost. So my, there it is. Okay. Now the, the complication then for a tutorial user is they may end or, or do you envision using a different volume for each tutorial? No, we, we did have a discussion about it. We should use this volume, the name volume or not. Since it uses, but we didn't decide. We, we didn't completely decide what we'll use in the end, but right now we are using. And that, that sounds great to me. I was just curious. If I'm not sure how users, if users tend to do one tutorial and then stop, or if they do two, two or more tutorials, and I was worried if they, if we have a single volume, is there a risk that the two tutorials might collide with each other by using the same volume? Yeah. So that's, that's very elegant. Nicely done. Congratulations. Congratulations. Anything else you'd like to let me put the link to your, to your repository here so that we've got it. We created a job, ran the job, watched it run on the agent. We installed the plugin, right? I confirmed it worked. Saw that the agent and the controller were Debian. Very nice. And so then you said that the next step, next target is the next sort of big target is using Docker inside somehow. Yes. And the like tutorials need Docker and Docker to work. So we'd have to do that. Yeah. So, and I, I am not, I am not clear how that, how you'll do that. But we know we have a working example. Well, sort of. We have a working example in the current tutorial, but it does not use an agent. Yes. We were thinking of creating Docker and Docker, but example, what we decided it will be, it is a security risk. So we'll put that towards the end, but for tutorial, we will need it. Exactly. See, that was, that was one of the things that I look forward to you exploring to see what alternatives are there for easy execution of Docker operations in an environment that's already running in Docker. And that's, that's the place where I, well, this horror right here, I'm delighted it works, but it is scary what it's doing, right? I mean, it's, it's doing, it's doing things that this Docker and Docker stuff is doing things that are, have security risks. Good. Very good. Anything else you want to, want to share with us? So next, this week and next week, I'll probably working to on the midterm presentation too. All right. Meg, I've got a question for you and for me then. So let's go to, I want to look at the Jenkins at the installation instructions for Docker. So installing Jenkins on Docker, if I remember correctly, it does a full Docker in Docker even here. Yes. Okay. Ashutosh, you'd already seen it. It does. Yeah. So, so we need the same, the same technique here as we need in the tutorials, because it's, this is also doing Docker in Docker. And it was doing it specifically because we wanted people to be able to use the same pattern to do this as they did tutorials. Good. Okay. Great. Nice. Nice. Needed for, needed by four tutorials. And the Docker install instructions. Good. Very, very good. Anything else, Ashutosh? No, nothing else. Okay. How about we take the next topic then. I'd made myself a promise that I'd get done by in about 20 minutes. We're a little over. Meg, if you're okay, I'd like to take the, just this one topic on the open pull requests of interest we've got. You remember Jeffrey Chen had submitted a pull request that you would evaluate, you had reviewed. Right. And mold wiki stuff, right? Exactly. It was a migration of two things, the administrator, some administrating material and the best practices page. And so I started working on it last weekend. And while working on it. Found that, Hey, I couldn't make changes to it. So I ultimately ended up taking Jeffrey's work. But I closed the pull request. So kept all of his commits, but closed the pull request and then split it into two pull requests. So because the original thing was targeting both administering Jenkins and best practices. The split into two was one half for administering Jenkins. And one half for best practices and the, the administering Jenkins piece. I felt like was well enough reviewed after your reviews and Kevin's reviews that earlier today during docs office hours, Europe, I merged it. All right. And so what we'll see is that Jeffrey's work here. And the things that I had added to it ultimately did end up in the book right here. Let's go look at it. So I think it's in, is it managing Jenkins configuring the system? Here's the Jenkins. Like, oh nice. Oh, and it's no longer a work in progress. Right. Yes. So, so a win there. Now the next step needs more text and that's this. This best practices thing because what I feel like is. Recommending best practices is a dangerous statement of opinion. Right. And we want to have good opinions before we go all the way. Right. So I took the text that was there and then I added a bunch of my own opinions. And of course that's dangerous. But that's what I want. And I had some places where I said, we need more words and Kevin said, we don't take that out. That's junk. And my answer was no, we really need more words there. Right. I have nothing against best practices. If you get a disagreement, then you can say some people recommend doing this because of this. Right. Other people it's very Talmudic, but, but it works. Well, but using a Talmudic approach, I think is the right approach. It's, hey, let's talk about the compromises because the word best in this case miss might be misperceived as the one true way. And that's not what it's describing. Right. It's describing for a particular set of circumstances. Here are a set of good practices that have worked for some people. And we've gotten the convention of calling a best practices and it's awful. And, you know, but it should be good practices. You're absolutely right. Yeah. Yeah, exactly. And a software testing community that I'm in. Loves to, to, to berate the use of the word best in that case, because it's a, it's what a superlative. It's, it's a maximal thing. And the usual answer is nonsense. They're not maximal. Right. But the cool thing about this one is that Darren has four or five videos that are now embedded in this. And so let's go look at it briefly to see how it looks in the, in the prototype because I think it's the right approach. Now, one of the problems with this approach is it's. Well, it's imperfect like anything. But here, if we look at using Jenkins best practices, what you'll see is a series of videos that are almost back to back with each other, GitHub organization folder, get lab group, uh-huh. Bit bucket team or project or giddy organization folder. Now I could separate each of them with a heading or with some text, but they are most, most of our users will have exactly one of these and no more. Right. And so having them choose now, maybe it would be better if there were subheadings here so that in the table of contents under organization folders, they could click. GitHub. Giddy. Bit bucket because that might get them closer navigation. Right. The other thing that I haven't seen these videos, but is there like do these, do you have two or three points that are like your summary takeaways? Cause the problem, one of the things that bugs me with the embedded videos is it's a 20 minute video and I watched it and I liked it. And there were a couple of things and it's like, what exactly did they say? And this is not fast forward as nicely as my Tivo does. Right. I've been here trying to jump around. It's like, where was that gem? If there are a couple of gems that could be out there or the other thing is summary commands that I could copy and paste. It's very hard to copy and paste out of a video. Good, good suggestion. And that would give us an excuse then to put headings and to put a brief two or three or four bullet summary of, of that video segment. And then same for this video segment and same for this video segment. Good idea. I like that. It's the stuff that I, after I've watched the video three times and consumed at the stuff, I want to go back to quickly and grab. Right, right. So what you're saying is there, it is far better to illustrate the hot spots with text, copyable text. Good. I like that. Okay. Yeah. Good suggestion. Now the others. Yeah. See, these were, these were, okay. Now we're really getting in Markweight opinion mode. And this is, this is, this is a dangerous world, right? But it's a good world. I do go up all pine all you want. Right. So what I started with was, hey, first choice you should use organization folders. They are the most efficient way to do it. It means you, you, you create one job and it will create multi-branch folders for every repository and jobs for every branch and massive automation. But there may be cases where multi-brand, where organization folders aren't quite what you need. Okay. You could then step down one level and do multi-branch pipelines. And it's, hey, if something prevents you from using organization folders, then at least use multi-branch pipelines. And this is where the needs more wisdom. Why is multi-branch pipelines not a link to the chapter, the page about multi-branch pipelines. Organization folders was. I'm not sure I understand the question. Percent. Wait a minute. Go, go back where you were. See organizational folders that in the line just had it. It ends, but help me. Oh, wait. Okay. Just above this one refer to the organization folders documentation for more details. Right. Now go down. Okay. So now you want me to go down here. To see this one where it says. There it says, if something prevents you from using organization folders, then use multi-branch pipelines. So I'm saying they are multi-branch pipelines should link to. Oh, oh, I see what you're saying. Okay. Good. All right. That's a, that's a good, good suggestion. I should probably follow the same pattern. Let's see. Well, let me think about that. Okay. Because the, what I did was since this sent the closing sentence had the hyperlink in it here. I put the closing sentence with the hyperlink here as well. But what you're saying is. I see. Okay. Now, but, but it may be that this wording is so weak right now. That's why you're even seeing this word at the end of this. That's right. I was at the end. I would say keep it in the back of your mind till you get the more words in and see what you've got. Right. Good. And also I would have, will those sections link back to this. They, they would generally not because best practices for me are a high level summary. And the documentation is a detail and there's not much gain for the user to jump back to the best practices stuff. Yeah. I don't know. I, I like lots of links. And some people complain about getting stuck in a circle. And I don't, because you never know where people are going to start. Right. And, and now, now this is one where, where it's, it might be worthwhile to consider. So this page, for instance, this is actually a page that Kevin Martin's detected was not in the, in the list on the left. It was not in the index on the left. Oh, so he moved it. He put it there correctly. Very nice. But it, it's rather older descriptions of some things. So for instance, it mentions GitHub organization or a BitBucket team, when in fact right now we have five or six organization folder implementations. GitHub, BitBucket, Giddy, GitLab, Tuleep, and I think maybe even one other. So, so we've got multiple implementations of these. And I think they're actually mentioned all the way at the bottom. Nope, nope. They aren't. There is a page that mentions which ones are full organization folder implementations. So, so this page may, may benefit some from at least some minor rewrite. Right. Okay. So that was the, that was the one I wanted to show. And I think I've, I think I've extracted the topics I needed from, from which is there are things to improve on the best practices page. And maybe one is a disclaimer describing that the words, the word best doesn't actually mean best. It means in some context for some people, these have been good practices that have worked for them reasonably well. Okay. So. All right. Well, any other topics I'd propose, given my level of weariness and Ashutosh, let you get some sleep as well. Oh no, it's start of your day. Get, let you have a good day today that we call today for done unless there's something else you wanted to go over, Meg. Nope. Sounds good to me. Ashutosh, anything else for you? No, sounds good to me. All right. Thanks to all of you. Good meeting. Good work by everybody. I'm impressed. Well done Ashutosh. Thanks very, very much. Thanks.


Welcome to another Adventures in Film. Today we've got T-Max 100. And as usual, when the roll's done, the adventure's over. My name is David Patton. When I started photography, I wanted to make art. I wanted to be a landscape photographer. But with a family to feed and bills to pay, I decided it would be better to be a working photographer than a starving artist. So I took a job as a photojournalist. 25 years and thousands of assignments later, it was time to go back to my first love. It was time to follow my passion. Come along on my journey to become the best black in my photography I can be. Whether it be film or digital, I will be sharing what I learned through my successes and my failures, in hopes to inspire and educate. This is my journey. This is Bright in the Edge. My adventure in the Oregon Coast Range. I'm on the East Rim Trail of Mary's Peak. I'm not sure what I'm going to find in these woods. I've had a dusting of snow. I don't know if I can use that for my favor or not. We will see. I'm not sure how far I'm going to go up this trail. I didn't expect this really to be seeing snow this low. Coast Range is typically pretty good year-round. We just got a little bit of snow last night. Not too bad to drive in. We're going to see what's up here. Hopefully I can find something to photograph. If I don't finish this roll of film today, the adventure continues tomorrow. I'm trying to figure out a composition here. It's fairly cluttered. I'm thinking about using this kind of one tree as an anchor point for the composition. I really like the dusting of snow on these leaves. I do like the backdrop of all these other trees. It helps to add a lot of depth to the image. I'm not sure if I can make a composition that has balance. I'm going to try to get myself in there without getting too wet. I think it looks pretty good. I think I'm going to open this up just a little bit to make sure I get the darker elements with enough detail. The light is pretty even right now. I'm going to overexpose this for about half-salve. I tend to overexpose just a little bit anyway with negative film. If you're interested in the settings, what I've got going here is about a two-second exposure between F-16 and F-22, kind of going back and forth between the two. I think I'm going to shoot a couple frames and bracket this one just a little bit because the light keeps changing pretty quickly. It all of a sudden got kind of dark in here, so I'm probably going to have to change my exposure on my next frame. While I'm looking for another shot, I thought I would have a little chat. I probably talked too much. That's part of the connecting with the artist. I still haven't figured this YouTube thing out. The main reason I started my YouTube channel in the first place was to connect with an audience that might find something in my work, in my photography that they like or are drawn to. I thought YouTube might be kind of a good way to see who this person is behind the artwork, behind the photography, trying to navigate through it. It's easy to get sucked in by the numbers for the lack of, in my case. Videos like this one won't get a lot of views. That's just the way it is. When I go out and make photos, those videos are starting to be watched. These are the kind of videos I like to make the most because they're really the best representation of who I am as a photographer right now. So I was thinking when I'm doing these videos to highlight who it is that I'm watching right now on YouTube. Since this is a YouTube thing. Who am I watching? Who's inspiring me to spend 10 to 20 minutes watching on YouTube? So from time to time, I'd like to highlight in my videos who I'm, or mention anyway, who I'm watching. I mean, my channel's so small, it's not like they need media. I mean, viewers are away. So today, I'm watching a gentleman in the UK named Steve. I can't, or Butcher's last name, Onion, Onion, Steve Onion. He has about 15,000 subscribers. Not a huge channel. I really connect with his work. I really seem to connect with him as an artist. He seems to be a pretty hard working photographer. He shoots a lot of film. He does digital as well. There are some of his sensibilities in photography line up with mine when it comes to things like sharpness. Although he tends to like his grain a little more than I do. That's okay. So, if you're wanting to be inspired and you want to see some really good film-oriented landscape videos, give them out, check them out. It might be worth your time to have a look. Woodland photography, man. It can be so challenging. Here's the scene right here. I got to make a photo. I don't know what you think. There's got to be a shot here. Kind of waiting for the light to even out just a little bit. Kind of some hotspots on some of these trees. I move forward about, probably about 10 feet from where I nearly saw it and looked around. It's just, it's like a cathedral.  I think the light's about right. I think I'm going to take the picture. I'm trying to find some detail shots with the snow on the foliage in a protected area where the wind's not too bad. I haven't found the shot just yet. I need to be able to isolate it from the background. I'm afraid that the snow is going to melt before I find my shot. I still haven't found my detail shot yet. I tracked it to this trail shot. Once again, I'm exploiting what this woods is offering. What's really offering me is a lot of really straight trees. So that's going to make up a lot of my compositions today. Something about being in the middle of these trees, these standard trees just really does it for me. This shot here is with a normal focal length. It's about 75mm on here. Compressing the trees just a little bit more than with a wide-angle lens. It's coming down pretty good now. It looks to be frozen, which is fine because I'd rather have that be frozen than get all wet. It looks more like ice pellets than sleep than it does snow. Well, it's all starting to look the same to me now. I'm not seeing a lot of compositions. It's just a little bit too much chaos in here. What I think I'm going to do is I'm getting kind of hungry and I'd like some coffee. I'm going to work my way back to the car and drive around a little bit. I still have a few shots in this roll. Let's see if I can find something on this road that I drove up. I passed a few potential shots on the way up. I'd like to save a couple of shots just to make sure I find something cool down there that I passed on the way up here. I'm going to end this video right here. The last frame has been shot and the adventure is over. This is the following morning when I started this video. It's been a lot of fun trying to get images to fill up this roll. It truly is an adventure every time. As I wrap up this video, I'm going to talk to Steve Onien's YouTube channel and subscribe, show him some love. He's working really hard over there. I'd like to see guys that are putting so much effort into their videos being rewarded for it. He's out in the field shooting with film. He's offering a lot of time, cost, expenses to bring some really good content to his subscribers and viewers. I'd love to see that effort rewarded. I'll head over and show him some love. For me, I'm going to wrap up and go home and get some coffee. Until next time, because you're coming along for the ride.


Hello people of the internet, my name is Johnny and welcome back to another video. I got it! I finally got it! The Twisted Ones Graphic Novel and let me just say, before we get too far into the intro, this is amazing. No words at all can describe how god dang amazing this graphic novel is. Please go buy it. This is, again, amazing.  Editing Johnny Box.  He said making this thing a possible. What I think he means is making this thing a possibility or making this thing possible. Who knows? He's kind of a nut job. Hopefully we can get the fourth closet to look as good as this does. Not at all trying to discredit the work from the first graphic novel by, I believe it was Pinky Pills. This is still, again, I still really like this graphic novel, but oh boy, I, oh boy. So thank you so much for clicking on this video, subscribe if you're new, boom, there's some analytics, maybe it'll encourage you to subscribe, hopefully. And smash the like button and let's talk about this book. The graphic novel of the Twisted Ones is now out. It came out yesterday on the second and as you can see right there, the Twisted Ones. That book was released a couple years ago. I don't remember exactly when, maybe like 2016-ish. And we finally have a graphic novel version of the book. And let me just say again, I can't describe how absolutely amazing this graphic novel is. The amount of work that was put into the illustrations is amazing. It truly is amazing what they managed to do here with this book. You can really see the emotions of the characters through the pages now. Which as someone who has not read the story of the Twisted Ones, basically ever since the book came out a couple years ago, it was, it was, it felt really nice to go through the book again and really see the emotions behind the characters now. Because when you're reading the book in your head, you can only picture what they're feeling, how they would react to certain scenarios. And this book really did that justice. You can really see the terror, the surprise, the enjoyment, the tenderness between certain characters if you know what I mean. And also like subtle changes, like clothing throughout the book. Right? I was looking through it just now and Jessica changed outfits briefly a couple of times. Just subtle changes though, but enough that they're noticeable. And like when John met up with Charlie while he was walking, he was wearing his you know construction uniform. That's good detail right there. The designs of the Twisted Ones are again amazing. The design of Springtrap, William Afton, David, whatever you want to call them, absolutely spectacular as well. Even the OG gang of Freddy Bonnie and Chica and Foxy, I nearly forgot Foxy right there, show up as well and they too look pretty good. I did notice however that the Let's Eat on Chica's bib was red, which I believe was something that appeared in the first graphic novel as well. So I don't know if this new artist was using the last one as reference. I just don't know. But like plenty of Endoskeleton show up in the graphic novel, they look amazing. The detail on them is spectacular. A couple of Toy show up like Ella and Theodore and Stanley, they too look amazing. And the backgrounds and the characters, oh dude it just looks so good. There's really not much else I can say besides this book was so good. If you have not read the Twisted Ones and usually don't like reading long books, get this.  Again, I'm not trying to discredit the work of Pinky who illustrated the first graphic novel. I'm just saying that I think for the good of the community, and again I'm trying not to sound like a giant jerk here, for the good of the community, this new art approach is definitely the way to go. I am honestly really looking forward to the fourth class at graphic novel now. Now granted that may take a little bit to release, probably like late next year, though that's not confirmed. We don't even know that they are making one, though I presume that they are, you know, I'm guessing that they will. Seeing as there are three books in the trilogy, you know, that's why it's a trilogy, and they only have one more left to do. I really hope they continue with this artist. I think they did an amazing job. Again, like I said at the beginning, I'm not going to show all the pages. I'm going to go through and I'll show a few pages up on screen while I talk, just because I want to get the point across just how good this graphic novel is.  Show the support for Scott, for the illustrator, for Scholastic, everyone that worked on the book to show them, like, this was a good graphic novel. It really is. And I've never really had to do this before, but I literally cannot put in towards how amazing this graphic novel is. So if you have a store near you that sells books, like A Barnes and Noble or some other book seller store, go see if they have this book. Go buy it. Even if you want to, you know, just reread the twisted ones, it's a nice condensed version of the book. As well as getting some great illustrations to go along with the story. Again, there's really not much else I can say. Everything is just beautiful. Honestly, I could like keep looking through this book. I can keep saying this book is really good. Go out and buy it. But I'd rather not make that the whole video, even though that's kind of what it is. I definitely do see myself going back and looking through this book all the time just because, again, it looks so good. And really that's all I can say, because that is just the truth. Really. That's, that's, that's it. It's just so good. So I'll try not to take too much of your time up with this video, because really all I'm saying is that it's good and go and buy it and making that into an eight minute video. Well, a good idea definitely is very cheap and tedious. So I'm sorry about that, but really that's all I can say. It's amazing. Please go buy it. Let's, let's show some support. So hopefully the, the fourth closet, the final book can be just as good as this one. And again, I mean no harm towards Pinky. I think, again, I really do like this original graphic novel, but the new art style is just chef's kiss. Yeah. I'm very excited to see where they go in the future with the graphic novels. If they're going to go back to Pinky, because if you didn't know, she was offered to illustrate for the Twisted Ones. But as it turns out, she just, she's having a baby. Congratulations, Pinky. If you're watching this. I think that's maybe why she declined and that's why Scott had to find new artists though. That's not confirmed. Yeah. I'm intrigued to see where they go from here on out. A lot of exciting videos coming out this week. Boys tomorrow. The return to FNAF A.R. It seems like we're getting some skins, unfortunately, but it's a lumix. What, what did we expect? And then, yeah, I'll cap out the week on Friday with a video talking about FNAF X fortnight because apparently a lot of people want to see that for some reason, even though not many people are excited about it. So I'm so confused if people want that video or not, but I'm going to make it anyways. So thanks so much for watching. Sorry for making an 8 minute video about a book saying it's good and go buy it. But really, again, that's all I can say. So thanks for watching and I'll see you all on the flip side. Goodbye.


So my name is Roger Gnazio. I'm a tech lead at Mesosphere specifically for the day two operations team. So everything that pretty much comes after provisioning logging, metrics debugging, that kind of thing. Today we're going to be talking a bit about the metric side of things. I'm also the author of Mesos in Action. It's more of an operations-focused book on Apache Mesos and it also goes into DCOS a little bit. So if you're interested in picking up a copy of that, you can use the code on the screen at Manning.com and get 39% off. So today I'm going to talk a little bit about what day two operations means in the context of Mesosphere and DCOS. I'm going to provide a quick introduction to the various Mesos APIs for monitoring. Then I'm going to introduce Snap, this cool new plugin-based metrics collection system. After that, I'll touch a little bit on what it was like to develop the Mesos plugin for Snap and how you can write your own plugin. I was going to give a demo, but it looks like because of the difficulties, I won't be able to do that today. So maybe I'll do something and post it online after the talk. So first I just wanted to start off with what day two operations means to us at Mesosphere. And we joke a little bit internally at Mesosphere that day two operations is everything that comes after day one. So what does that mean? Everything that happens after you provision a Mesos or a DCOS cluster falls into day two operations. So that's logging, that's debugging, that's metrics collection. Really anything that you need to do to operate and ensure the health of a cluster. We recently started a day two operations working group in the DCOS community Slack. So this isn't just a team at Mesosphere anymore. This is I think 30 or 40 people in the DCOS community Slack that are interested in this kind of work. So we would love it if you joined us. You can join the day two ops channel on Slack or we have a mailing list day two ops working group at DCOS.io So before I get into the snap portion of this talk, I just want to just want to introduce the various Mesos metrics APIs that we have available. These APIs provide information about workloads running on the cluster, individual containers, frameworks, resources that are being utilized, etc. So there are a lot of these different API endpoints and I just want to highlight a couple here just for the purposes of this talk. The first one is the redirect endpoint. In a Mesos cluster, you commonly have somewhere between three or five or seven masters for a production highly available deployment. The redirect endpoint returns a HTTP 307 redirect to the leading master. And this is important because you never want to query the non-leading master for its metrics. I'm not sure if this has been improved in recent versions of Mesos, but it used to be if you quoted a non-leading master, it would return information, but it might not always be up to date. So the leader is always going to have the latest state of the world of your cluster. The next is this metrics snapshot endpoint, and that's a summary of the master's metrics. Kind of a high-level in operations view. It's things like how long it's taking to query the Mesos internal registry and how many messages are being sent back and forth between the frameworks. There's also these state and state summary endpoints and this provides a high-level or a detailed view of all of the cluster state. That's all of your frameworks, distributed systems, containers, configuration flags, etc. Now that was just on the master. So that's the three or the five or the seven machines that make up the master quorum of your cluster. On the agent, on port 5051 we have a few different endpoints. Specifically, we have this new containers endpoint that I believe came out in either Mesos 1.0 or maybe .28, and this provides metrics about running containers, container IDs, CPU, memory, disk usage, etc. This used to be available at the executor level, but now you can query it for individual containers if you're looking for that level. Just like on the master, the metrics snapshot endpoint provides the agent metrics. That's the number of tasks and what state they're in, whether it's running or failed or staging. The used CPUs, the allocated memory, etc. So there's a lot of information available there. All of this is returned via JSON over HTTP and then state is the detailed information about the node. So on a specific agent, what frameworks are currently running, tasks on that agent, what labels are associated with those tasks, what configuration flags are present for the agent, and so on. But this was just getting to the tip of the iceberg. There are dozens of different API endpoints available. I don't have time to cover all of them, and not all of them are necessarily relevant for this talk. One of the endpoints that's available on both the masters and the agents is slash help, and this help endpoint details all of the available endpoints and a little bit information about what they're used for and what response you can expect. So you can query those on either the master and the agent, four-year specific version of Mesos, or if you go to mesos.apache.org, you can see the documentation for the latest version of Mesos. So that's just an overview of the information that we have available, and that information could be very valuable depending on what you're trying to do in your company or with your cluster. So, well, I'm just going to get right into SNAP, and really the best way to describe SNAP is from its project website. SNAP is an open telemetry framework designed to simplify the collection, processing, and publishing of system data through a single API, and that sounds great, but what does that mean in practice? It's a plugin-based architecture for collecting, processing, decorating, and ultimately publishing metrics through to an end system, whether that's something like ChirisDB or InfluxDB or some other time series database, and it really allows you to decouple the publishing of metrics and the collection of metrics, and some of the processing that you might want to do, like aggregating or taking the moving average of specific values and so on. This is pretty powerful because you could have you've really decoupled the publishing of those metrics away from the collection, and SNAP is there to kind of handle all of the translation between collection, processing, and publishing, and the SNAP workflow, just borrowing these images from the website, looks a little bit like this, where you can have varying workloads, where you're collecting all of these different heterogeneous metrics, you can filter, you can add context, you can tag metrics, you can aggregate them, and then ultimately publish them either onto a message queue or to a time series database, and you can visualize them with your tools of choice. Now all of this is happening really on each of the individual agents, so the processing for all of this is distributed. Like I touched on, it's a plugin-based architecture. These collectors and processors and publishers are running on each node. The really cool thing is that all of this is done with these dynamic updates. So there's no need to restart the SNAP daemon if you want to make a configuration change, or if you want to upgrade a plugin. You can do all of this without losing data. There's also a feature called SNAP Tribe, and this is a GASA-based protocol for handling configuration changes. So if you were to push out a configuration change to a hundred or a thousand node mesos cluster, to collect different metrics, to change the polling interval, and so on, you're able to do that without having to use a configuration management tool or without having to log into each one of those machines. SNAP handles that for you. So that's pretty powerful if you're using something like CoroS that's minimal by default. To control SNAP, there is a command line tool called SNAP CTL. There's also a REST API, and the plugins use the REST API for communicating with the daemon. I believe you can also use the REST API directly, although most people just use the command line interface. Now, a plugin is implemented in Go, or anything that understands Go's GRPC protocol, and I guess at a high level, a collector plugin looks something like this, where you have to implement these three methods. So there's the configuration policy, there's the metric types, which actually, when the plugin is loaded, getMetricTypes is called, and it populates the available metrics on that machine so that you can then build a task in querying that machine for its metrics. So without knowing the available metrics, there's really no way to know what you can collect. And then finally, there's collectMetrics, and this is called bySNAP on the interval, and it just runs through whatever code that you might need to collect metrics from the system that you're querying. So in Mesa, this would be hitting those various APIs that I mentioned, and running that on the interval that the SNAP daemon has set. So at a high level, that's what it requires to actually implement a plugin. So if we take a look at the SNAP collector plugin for Mesa's, which is available on GitHub, it's open source, we would love contributions. It looks a little bit like this. So just to break it down a little bit, we have one Mesa's plugin that handles both masters and agents. So the first thing you'll notice in collectMetrics is we determine if we are on a master or an agent, or if potentially they're both running on the same machine for development purposes. Next, we determine if the master is actually the leader. Like I said before, we don't want to query a non-leading master. If the master isn't the leader, we just don't do anything. We get information about the frameworks, about the running containers, and so on. We put all of this into a metrics array, and then we return it to the SNAP daemon. From there, we can take this collector plugin. Those metrics can be, depending on the task that we've defined, that the workflow, we can actually push those through a processor or directly into a publisher and off to a time series database. In the Mesa's plugin at present, these are the various namespaces that we have available. There's the master and the agent namespaces, along with these asterisks. The single asterisks are referred to as dynamic namespaces, so they can be built dynamically based on the underlying service that you're querying. I'm not going to read everything from this slide, but essentially what we can do here is these dynamic namespaces can be filled in with the framework IDs and the executor IDs, so that we're able to associate metrics with individual workloads, and that way you can determine how much memory or CPU marathon is using, versus how much Chronos is using, versus how much Jenkins is using, and so on, and really get a good snapshot of what your cluster is doing and who's using it. Deployment looks a little bit like this with the Mesa's master and Mesa's agent running on each of these nodes. The snap daemon is also running on these nodes, and you'll see on the standby nodes the daemon is running, and it's attempting to collect every, let's say, 10 seconds, but not necessarily doing anything if one of those machines is not the leader. And to actually configure snap, you have this concept of a task. The task contains the metrics that you want to collect, where you want to publish them to, the hostname, or the IP address of your time series database, the IP address in port that Mesa is listening on, and so on. And from here, like I said, each of these is running on the node. You need a way to get those files onto the node, but you could use snaptrod to roll out configuration changes. And then all of this is rolled into, you know, some time series database, or an HTTP API, or some other publisher plugin that you can develop, or use one of the publisher plugins that's already available in the snap ecosystem. This is a little bit hard to see. Like I said, I couldn't do the demo today because of technical difficulties, but this is just one shot of a Grafana dashboard that I created quickly that shows all of these metrics being plotted. Up at the top, we have cluster health, the number of tasks in the various states that they're in. In the middle, we have cluster utilization overall. This is how much memory and disk and CPU is available on the cluster and how much is being used. And then we have resource utilization by framework, and that's, I think, Marathon is the only framework running here, and it's using somewhere around 8 gigabytes of memory. Cut off a little bit here, you can see maybe down at the bottom, there's also resource utilization per executor. So if you have a misbehaving process or application, or you want to determine where your network traffic is going on a specific host, you can build dashboards and you can query something like influxdb for all of those metrics over time. And like I said, I really wanted to give a demo today. Unfortunately, it's just not going to work out, so I do apologize about that. But if you'd like more information, this GitHub repository contains a vagrant file, and in the vagrant environment, you can pretty much just do a vagrant up, it will provision a single-node mesos cluster with a snap with the snap mesos plug-in with influxdb, with Grafana. Pretty much everything that I wanted to demo today is available in this repo. You can just run vagrant up, assuming you have vagrant, and be able to get it. Snap is also available, it's open-source, it's under the Apache license. You can get that and check out the various different plug-ins. You can also check out snaptelemetry.io, there's a whole plug-in catalog there with all of the different collector and processor and publisher plug-ins. At MesoSphere, we're also working on a metrics project for DCOS, but a lot of this has been Meso specific, and it also applies to DCOS. We're working on a few different things if you want to go check out the DCOS metrics project. And like I mentioned earlier, we'd love for you to join us on Slack in the Day2Obs channel, or on our mailing list at Day2ObsWorkingGroup at DCOS.io. Because I didn't have time for the demo, I'd like to open it up for questions if anyone has any, or if you want to send me an email at rogeratmesosphere.com, I'd gladly accept it. I don't see any hands, so we're going to end this talk a little bit short. Thank you for attending, and enjoy the rest of the conference. Thank you.


This video is going to be divided into three parts. First, is a high-level intuition on what K-means is, what it does, and the algorithm. Next, we're going to represent it in math notation. And finally, use this notation to code an image compressor. I'm AJ Hathor, and let's just jump into the big picture. K-means is a non-parametric method of clustering. Non-parametric? That means computation complexity depends on the number of samples. Clustering? A group of similar data points to form clusters. How many clusters? Well, how many ever you want. This is defined by setting K in K-means. It's a tunable hyperparameter. So what's a hyperparameter? A parameter that is set by the programmer manually, and it's not learned by the algorithm. In summary, K-means is a non-parametric method of clustering where we pre-define the number of clusters. You got that? I know you got that. Now let's move to the algorithm. Step one, determine the number of clusters K you want to divide the data. Step two, set K random non-overlapping points as the cluster centers. You can just take these from your data points. Step three, assign each of the end data points to the closest cluster center. Step four, for every cluster, compute the centroid. Step five, the centroid is now the new cluster center. And then for step six, we just go back to step three where we repeated until convergence. So how do we define convergence? Well, after successive iterations of the algorithm, the cluster centers don't change that much. Then you know that the algorithm has converged. We'll end up with the end data points separated in K-distinct clusters. So here's a question. When we're computing the cluster center, why exactly are we considering the centroid? Well, in order to understand that, we'll have to go through the math and let's do that right now. As with any type of statistical learning method that we've discussed before, the idea is to optimize some objective or minimize a loss. For K-means, this loss is the distortion function. In plain English, this distortion function is the sum of distances of every sample from its cluster center. In this form, J is the distortion measure. Xn is the n-th sample. Mu K is the k-th cluster center. And R represents the cluster membership. Here it's an n cross k matrix with values either 0 or 1. R and k is equal to 1 for only that value of k, where the k-th cluster center is the closest. Otherwise, it's 0. So you can see that every row is actually a one-hot encoded vector. In k-means clustering, the only set of parameters we need to learn are the cluster centers. The idea is to find the optimal cluster centers, Mu K, that minimize the distortion. We determine this by taking the derivative of distortion with respect to Mu K and equating it to 0. Let's split the sigma over the terms. Now remove the common 2. Take Mu K out of the sigma as it is independent of n. And bring all the terms to the right-hand side. In English, what does this represent? Well, the numerator is the sum of all samples Xn belonging to the k-th cluster, while the denominator is just the number of samples in the cluster k. This sum of all samples in a cluster divided by the number of samples in the cluster is the definition of centroid. I hope you now understand why we are determining centroids in order to define the cluster centers. It actually gives us meaningful results. Now let's revisit the k-means algorithm, but we'll map it up a bit. Step 1, we'll initialize some values for the cluster centers Mu K. This is typically k-discreed data points. Step 2, we'll set the initial distortion to infinity. Step 3, determine the cluster membership RnK for all samples. Step 4, determine the new distortion J. Step 5, recompute the cluster centers Mu K with the centroid update rule. And for step 6, we go back to step 3 where we once again determine the cluster membership. We repeat these steps until convergence. That is, either until the cluster centers don't change between successive iterations or if the distortion J decreases ever so slightly in subsequent iterations. Let's now use k-means to build an image compressor. Take a look at these baboon images. They don't look too different, right? Well, they kinda are. The one on the left has hundreds of colors, but the one on the right has just 16. We can see this difference if we zoom in. On the left, we see a nice gradient while the image on the right is kinda choppy. The idea behind compressing an image is to replace every pixel in the image with one of k-colors. These k-colors should be chosen such that the distortion of every pixel is minimum and determining these k-colors is done with the k-means algorithm. Effectively, the size of the image is decreased significantly. Given just 16 colors, each color can now be represented in 4 bits as opposed to the 24-bit or 32-bit color. First, import numpy, Python's library for math and matrix operations. Create a class k-means with a constructor that takes 3 arguments. The first is the number of clusters that is k and k-means. The second is the maximum number of iterations to consider returning. And the third is the tolerance. This is the difference between successive distortion values to be considered converged. Next, define the fit function which does the heavy lifting. It takes the n cross d input matrix as an argument. Now, just follow the exact steps we described in the k-means algorithm. Initialize mu k as random samples. Initialize distortion j to infinity. Compute the cluster membership r. Compute the difference between current and previous values of distortion j. If it is less than the tolerance, then the algorithm is converged. Otherwise, update the cluster centers and continue. And we're done here. Now, we can build the image compressor around it. We define a function transformImage that takes the original image as input and outputs a compressed image of code vectors, the k-colors we need to determine. Iterate over every pixel and replace it with its closest code vector. In the k-means image compression method, we read the image, call k-means to learn the best code vectors, compress the image using these code vectors, and save the new compressed image. And we're done. Just run k-means test.py and it will generate the compressed image in the plots folder. And so we have successfully implemented an image compressor using the simple k-means algorithm. You can get all this code on GitHub, so check it out. And that's all I have for you now. If you like the video, hit that like button. If you like videos like this, well, hit that subscribe button for videos on AI, machine learning, data sciences, and deep learning. For notifications when I upload, ring that little bell icon. Links to the code and other resources used in this video will be down in the description below, so check that out. Still haven't gotten your daily dose of AI? Well, click or tap one of the videos right there for another awesome video. And I will see you in the next one. Bye.