Good morning everybody. My name is Fabian Levy. I'm deputy mayor for communications for the city of New York Thank you for joining us today To update New Yorkers about the heavy rainfall and Flooding taking place across city and across New York City and parts of our state Our teams have been working diligently across city agencies and with our state partners to prepare them for the storm and protect our city Keeping New Yorkers safe and informed is our top priority, which is why we've brought together key leaders To brief the public this morning. We're joined by New York City mayor Eric Adams governor Kathy Hockel DEP commissioner writ Agrawala Nice and commissioner Zach Iskall as well as MTA CEO Jannu Lieber We're also joined this morning by NYPD chief of department Jeffrey Madry FDNY Commissioner Laura Kavanaugh DOE Chancellor David Banks New York City Health and Hospital CEO Mitch Katz Department of Transportation Commissioner Yadonis Rodriguez and senior vice president for corporate affairs at Con Edison Jen Hensley So without any further delay Mayor Adams, thank you. Thanks so much of Fabian and the entire team and New Yorkers for understanding How this rain condition that we are experiencing is something that we cannot take lightly and we have not taken Lightly this morning. I was out visiting East New York Flatbush, Karnasi and Sheepshead Bay to get a first-hand look at the impact of the rainfall and What it is doing to moving around the city Spoke with Governor Hockel who's here with us today. I want to thank you governor for once again Responding immediately and seeking out what it would ever help we need as a city and I really appreciate Communicating with you this morning. I am issuing a state of an emergency for New York City Based on the weather conditions, and I want to say to all New Yorkers. This is time for height and alertness and extreme caution If you are home stay home if you are at work or school Shelter in place for now Some of our subways are flooded and it's extremely difficult to move around the city Many of our area airports are experiencing delays And if you are out and encounter a flood a flooded area Be it on a roadway or a subway station do not enter take necessary precaution This is a dangerous Weather condition and it is not over and I don't want those gaps and heavy rain to give the appearance that it is over It is not we could possibly see eight inches of rain before the day is over City has already taken action to protect all New Yorkers including Rescuing those in need of help There was an issue on the belt Parkway by the Ocean Parkway exit Those are the types of roadway conditions. We're asking New Yorkers to avoid every single one of our agencies Has an emergency plan, and we are executing those plans We're ready, and you should be at well to be prepared for this moment Commissioner is cool from New York City emergency management will provide more detail updates To us on the situation throughout the city This is a time for caution, but it's also a time for community check on neighbors Do whatever is possible to unclog drain areas to allow the water to flow freely You would be surprised how if we remove leaves and of the trash from those areas It will really assist in getting water off our streets Check on your friends your relatives and especially those who are most vulnerable Such as the elderly and individuals with health conditions If the conditions are safe take time as I indicated get to clean out the debris from your drains particularly in between the Heavy rainfall and for the most accurate and timely updates sign up for notify NYC This is a moment to remind everyone of how important important notify NYC is those alerts go a long way To keep you ahead of what is happening. You could dial 311 or visit NYC dot gov Slash notify to subscribe. That's NYC Dot gov slash notify to subscribe, and you can also call 311 Being informed is the first step toward Ensuring their well-being of you and your family and your fellow New Yorkers and again I want to thank the entire team for their quick response And understanding that this is an extreme emergency Measures extreme emergency condition that we must be prepared for I want to turn it over to Governor Hoku Governor again, thanks so much. It was good talking to you this morning and your quick response to what is happening in our city area Really appreciate it Well, thank you mayor and once again I commend your leadership and the coordination between your team and our team at the state level It is seamless and the bottom line is we're here to help New Yorkers get through what is a a life threatening rainfall event And I agree with you that people should be able to stay home if possible but I know right now there's a lot of anxious parents wondering if they're going to be able to get their children home from schools and I'm working very close with Janet Lieber who'll give updates from the MTA but our priority once the immediate In the after immediate aftermath of this first wave of the storm and again it could come back again It could reemerge later again this afternoon But we want to make sure we get the subways the trains our communication System our transportation system up working because there's children who use the subways to get home from school People need to be able to know if they can get home from work And so that is priority number one to make sure that our subways and our rail systems are safe There have been significant disruptions without a doubt as truly heading north I just spoke to the county executive George Latimer of Westchester County The epicenter has shifted from the city regions very much so from Brooklyn and Queens and Manhattan where I spoke to the borough presidents to offer our assistance in the Bronx But is now heading north into the Hudson Valley. So we're also coordinating with those localities as well as Nassau County I will say this if people decide to venture out in a vehicle They do so at their own peril because even six inches of rain one foot of rain. It may look pretty innocuous It's safe, but that is a condition where your vehicle can be swept away And we lose more lives of people during flooding events of which we've had many especially this summer in the city and the Hudson Valley in particular The reason people lose their lives in a flood event more often than not is they're swept away in their vehicle So this is a choice. She will make we encourage them not to decide to do that. Please Stay home. Be safe. We'll we are deploying more buses We have more bus operators to be able to create options if we don't get the subways back on schedule the airports Right now terminal a at LaGuardia is closed because of flooding at a fuel field right now We brought in more resources from JFK to assist with that as well The flights are going in and out of the airports, but they are delayed and so people should check on The websites for that as well. Now. Also, we're trying to be as helpful as we can the city mayor You've been fantastic and we've deployed National Guard We've deployed a more pumps from our surplus to help you assist and everyone remembers what happened I was literally governor one week when Hurricane Ida hit and we walked those streets of Queens and many parts of the city and people literally drowned in their basement homes And so people also need to not get in vehicles But leave your home if you're starting to see water accumulate Don't wait until it's up to your knees or higher By then it could be a barrier to getting able to have access safely out the door as we saw before And so people really need to be taking this extremely seriously The state is there to help and I'll let we'll get through this together as we always do So thank you mayor for your coordination your outreach and uh look forward to continuing to work as we get through this event united as one Thank you very much governor huckle. Thank you mayor adams for the update Next we'll get a full update from new york city emergency management commissioner zack iskal Uh, thank you. Thank you, mr. Mayor. Thank you governor huckle First and foremost, I also want to express my heartfelt gratitude Uh to the emergency managers to our first responders our agency partners who've all been working tirelessly overnight and over the past day Preparing the city and and doing all they can to keep new yorkers safe They really are the backbone of of the city Our emergency operation center has been active and an operation to effectively coordinate our multi agency response to this ongoing weather crisis We have teams right now that are specifically focused on highway flooding Basement flooding and the impacts of the mta We're monitoring continuously a number of high-risk areas In order to deploy teams from different agencies to take immediate action as needed Uh through the day yesterday and today We've also been issuing really important safety messages and updates to the public It's crucial for everyone to stay tuned to the latest information as the mayor said that's the first line of defense Uh, we're also very fortunate. The national weather service is on site at our command center here at new york city emergency management They've been invaluable in providing real-time updates Allowing us to make sure that information is getting out to agency partners and the public with the most accurate forecasts And weather information as quickly as possible As I said, we're also in a very active response mode right now And we're confronting the severe weather conditions that are affecting our city as we speak A national weather service flood watch is in effect And let me be clear. We're taking this extremely extremely seriously Uh consultations with the national weather service began as early as wednesday They remain ongoing and we will continue to monitor this situation as it evolves This proactive approach is part of our flash flood plan Which is our comprehensive strategy that we've been honing over the years To make sure that we're able to respond effectively to situations just like this one I think it's worth mentioning that according to data from central park Today is the wettest day we've had since ita swept through the city a few years ago The governor was just talking about that event a week into her her tenure That's not a statistic to take lightly. It highlights just how crucial it is for all of us to pay close attention to the weather advisories And to always take the necessary precautions Uh, particularly in in brooklyn today and in parts of queens. We've seen significant flooding We expect another two to four inches of rain likely throughout this afternoon If you live in these boroughs or in a basement apartment or floodplain area Please make sure you have plans and are prepared to move to higher ground Your safety is our foremost concern We have multiple contingency plans in place, but ultimately, uh, You know, you are your first line of defense for yourself and your loved ones And that means acting on the information that we're able to provide Through things like notify nyc that the mayor mentioned um Also, uh, I urge everyone to extreme to exercise extreme caution Avoid flooded roadways If you are driving and I do not encourage you to be driving As the governor said even six inches of water can be very very dangerous Please take extreme caution particularly During high tides today throughout the city Um, we're also in constant communication with our law enforcement partners other agencies and community organization We're all working in lockstep to ensure your safety To that end, I do encourage everyone to sign up for notify nyc Mr. Mayor said, uh, you can sign up by going to nyc.gov backslash notify You can sign up by calling 311 or by downloading the app in both the google and the apple App stores it's available in 14 languages including american sign language And then as we start to shift into the recovery phase of this operation After the rain ends This evening and early tomorrow If you have damage You can now report damage to your home or your business as part of the city's damage reporting portal This is really easy to do. You can do report damage by calling 311 Or by visiting report damage dot nyc.gov So new yorkers, you know, we have a great history of coming together when it counts. We're going to do it again today Make sure to check in on your neighbors your loved ones, especially the most vulnerable Please pay a close attention to those alerts and advisories. They are designed to keep you safe. Thank you so much and stay safe Thank you very much commissioner iskall. Next we'll hear from new york city department of environmental protection commissioner red alcohol commissioner Good morning. Um just uh Just to build on commissioner iskall's points, um about the intensity of the rainfall, uh, You know, one of the things I think new yorkers should bear in mind is that Uh, when we see flooding we see it primarily due to the intensity of the rainfall in a certain period of time rather than the full duration Um, and as I think many new yorkers observed this rainfall started last night DEP started preparing for it yesterday around midday starting around 2 p.m. We were encouraging people to deploy the Flood barriers that DEP has been distributing to the most vulnerable new yorkers and we prepared by having Lots of crews on hand for today, which we knew would be busy The storm picked up significantly soon after 7 a.m. And I think new yorkers should be aware that between 8 and 9 a.m The brooklyn navy yard received 2.58 inches of rain in one hour Uh, and I think many of you know um in the aftermath of hurricane idah We were all educated about the fact that our sewer system was designed For 1.75 inches per hour And so it's no no surprise unfortunately as a result That that part of brooklyn and a couple of other particularly parts of brooklyn have borne the brunt of this The good news is we have a partially completed flood net system where we are able to monitor flooding on the roads In a couple of places that is en route to be completed over the next 2 to 3 years But right now what we have seen is that as the rain began to subside after 9 o'clock most many of those Flooding areas have begun to recede overall as we know This this changing weather pattern Is the result of climate change and the sad reality is our climate is changing faster than our infrastructure can respond We have been working at the mayor's direction For well over a year on a significant medium term approach to increasing green infrastructure and expanding the sewer system But changing infrastructure takes time. It's one of the reasons for the for more than a year We've been doing things like distributing flood barriers and helping people understand how they can protect their own homes and properties and working with emergency management under commissioner's leadership To recognize that the weather can kill you and that we have to respond appropriately I appreciate very much our collaboration not only with my city colleagues But also of course with the governor and her team and we are in close touch with our state colleagues. Thank you Thank you commissioner Next we'll hear from mta ceo jano lieber before we open it up to q&a. See you Thank you Listen, this is a tough travel day There are significant portions of the subway system that are shut down We are starting the process of reactivating certain lines But when water covers the electrified third rail, we have to do inspections So that that will be unfolding slowly We do have a full bus system operation today There are about 3,500 buses out there and that's especially important as the governor says to get kids home from school So if you must travel and heed the the warnings of the governor and the mayor and others That it's not a day to travel if you don't have to but if you do buses are available We have full on island railroad service But metro north at this time has really been shut down all three trunk lines of metro north newhaven Harlem and hudson lions are shut down at this time because of water in the south bronx And we are developing a service plan hoping that we can That we can reactivate those those operations We're developing a limited service plan for this afternoon with the hope and expectation We'll be able to get everybody home The good news is that new yorkers heeded governor hawkel's warnings and the mta's warnings and the city's warnings From yesterday and we had it is a friday. It's a lighter than normal commute It's always a light day on friday and new yorkers listen One of the reasons that they listened is that we have really a strong text alert system And I urge everybody who doesn't have it to sign up for mta.info Slash notify In the meantime, if you have to travel if you are still out there, please consult the mta apps My mta the train time app and the mta website mta.info We have full up-to-date information as I said about half of the subway system Is either fully suspended or partially suspended But we are starting to to the process of trying to get some of those lines back the other positive news I would emphasize is that since hurricane i to two years ago Governor hawkel instructed us to work with the city to try to make some of the stations more resilient So we wouldn't have those impacts on customers that you saw a couple years ago That collaboration has produced results and to date there haven't been any of those crazy Washouts inside the stations though. As I said service has been affected. Please stay close to those mta apps for real-time information my mta the train time app and www mta.info. Thank you Thank you CEO Lieber and so with that we'll open it up to some on-topic questions Okay, just a reminder if you have a question To use the hand raised function on your screen We'll start with Ethan stark miller from am new york. Ethan. You can now unmute your line Can you hear me? Yeah, we hear you Um, okay, so question for mr. Mayor. I mean there are asylum seekers in shelters that are tent like facilities that creed more On randall's island Are they safe in those facilities with these conditions and is the city taking any steps to move them elsewhere? We have no reported conditions in those areas on randalls and in Decreta mall location and of course if there are any conditions That are dangerous We're going to make sure that people receive the proper care and make the proper movement Mayor I just got an update on randall specifically We've seen very minimal leaking at randalls island and any problems that will be very quickly addressed So we have no issues really at randalls island at this point Great next we'll go to katie honan from the city katie. You can now mute your line The question's briefly chancellor banks. There have been reports of schools flooding I saw someone tweet that the cafeteria was flooded students can't eat My question for you and I guess the question for the mayor is We've now new york city has been flooded for hours now. We're finally now hearing from the city press release I think for a lot of people doesn't count. Why did you not take more precautions when it came to schools? Students don't know how they're going to get home if the subways don't return service And and secondly, I don't know if there's any disruption with bus services And so I guess the question is for the chancellor Can you give an update on schools and which ones are flooded and for the mayor? Why did you not speak to new workers sooner about what is a very very serious flooding situation that we knew about for At least 24 hours Katie we're the chancellor is going to go into the school situation and what we have right now But we have number one notified nyc and using the various social media channels and Commissioner isko has been speaking about this from Afternoon yesterday. So all the necessary precautions were taken. We have gone through these flood related and heavy rain conditions before And we followed the right protocol and really take my hat off to commissioner hisko and the entire team as they put in place a situation Of communication using the virtual methods and monitoring the situation Really doing what needed to be done and making the right notifications and informations and constantly keeping new yorkers up to date Chancellor, can you go into the school situation? Where where are we at this time? Sure, ma'am. Uh, first of all, it's just important to note that all of our schools every single one of our schools are open Uh, we're actively monitoring the situation across all of our buildings Um, we uh, we have safety plans in place and that's important to note That and we have folks in our schools who are trained annually to prepare for days Just like this, um our building response teams They fully activated and our protocol is to in fact shelter in place Now we do have a portion of our schools approximately 150 out of the more than 1400 schools that we have That have in fact taken on some water as presented some challenge But nothing has impacted the ability for us to safely educate our students in any of those schools Our schools have safety plans in place And uh, and they are moving forward Uh, uh, and they're keeping us abreast the stuff is happening In real in real time Um, there's one school is actually evacuating currently due to a smoking boiler And that is ps 132 in brooklyn and it looks like there may have been some water that got into a boiler So we evacuated in that school to is 78 which is located three blocks away Um, that's the only school that we've had to uh to date to date. Uh at this time have to actually Evacuate um our facilities and the school construction authority are working together to quickly repair Any damage, uh, even now we've got folks out and many of these buildings in real time Uh, now, also, I want to note that we received multiple media inquiries about The davis the david booty school which is is 228 in brooklyn situation And the message that the prince would have school sent Was premature in telling parents to come up to get their kids, which was precisely the wrong thing to do The conditions are worse outside of the schools And so we'll be addressing that but as the rain lightens up We're continuing to monitor the situation And uh, what's also important is that our buses are preposition for dismissal We've been in touch with all of our vendors And their preposition means That they are staging early in order to have enough time to get our kids home These are high axle vehicles And they'll be able to navigate any water in the roadways So, uh, in closing, I would simply say, uh, our schools are open. We feel good We're fully prepared. We're ready to go. We have taken on some water But nothing that has created an infrastructure problem where we our kids are not safe our kids are safe And we are we're continuing to monitor the situation Thank you chancellor and just to add a little to what the mayor said earlier, uh, our first travel, uh, Or excuse me our first notify nyc alert went out yesterday afternoon around 2 50 p.m We one of the first, uh, efforts we took was issuing a travel advisory We've been in touch with local elected officials across the city We've also been working with nonprofits And like the mayor said, uh, commissioner iskall was out there doing interviews yesterday uh, commissioner iskall is a one of our top, uh, Spokesperson spokespeople for this administration. And so, uh, he speaks on behalf of the mayor every single day And commissioner iskall could you share a few words about our decision-making process for notifying people? And how we make sure that new yorkers take these notifications seriously by issuing them, uh, thoughtfully Yeah, so first off so notify nyc as I said it's available in 14 languages including american sign language Uh, we do a lot of programs, uh, especially since ita to really encourage new yorkers to sign up We've done a number of day of actions. We did one about a month ago in jackson heights and across the city Encouraging people not only to sign up for notify nyc, but to make sure they have the information they need For these types of flooding events including the distribution of flood alarms that people can then install in their homes In terms of notify nyc itself for these types of events Uh, we take information, you know, we work very very closely with the national weather service We take their alerts their warnings their analysis and we make sure that we then translate that Uh, literally to 14 different languages in american sign language But also, uh, so that the public can understand what that threat might be In addition to making sure we're providing information not just what the hazard is But the actions people can take to keep themselves their loved ones safe Thank you commissioner iskall For the next question, we'll go to juliet papa's juliet. You can now unmute your line juliet you there All right, we'll move to can you hear me? Yeah, we hear you now go ahead Yes, thank you. Uh, if the fire commissioner was on this call, I was wondering or who can answer this Uh, have there been basement apartment rescues? And if you have any sense of where and how many because ida there were fatalities at that time And uh, I was wondering if there are any fatalities so far today Hi, juliet. It's uh, commissioner cavanaugh. We have not had any fatalities today We have had a number of calls for basements People trapped in cars As you've probably seen from the coverage you can see our vehicles Out there all over the city, but we have not had any critical patients or fatalities yet today Just a follow-up on commissioner cavanaugh In the since the last update I received about an hour ago We've had reports for six basement apartments flooded and we've had successful rescues at each of those Thanks to add to that as the mayor had mentioned We were made aware yesterday through oem about this storm and we put additional high axle vehicles out Into our operations. So we had equipment. I was specifically available and ready for these types of rescues Next we'll go to kelly menna from new york one kelly. You can now unmute your line. Oh, hello. Okay My first question is to our banks. I'm wondering why the decision wasn't made to go remote today for students If it's such a severe day I mean parents have to travel to get their kids kids have to travel to schools We've heard from multiple officials to stay off the roads today And then to follow up on katie's question mayor adams. It's noon right now Why wasn't the decision to call a state of emergency at 8am when we knew that rainfall was getting intense First and the chancellor could talk about remote, but we should be clear That we have only a certain number of school days that we could utilize And we must make sure we meet that and as you see The decision was was the right decision. We do not have any issues dangerous issues at our schools Our children are in the schools. They're properly being educated And I believe the chancellor made the right call And if there was a need to Close the schools, he would have made that call. But this was the right call Our our children are safe in schools. It's stabilized the family They don't have a level of uncertainty of how to get to and from their office spaces There is a big inconvenience when you close the school and parents disrupt their normal Workflow and we disrupt a child's educations flow at the same time This was the right decision to do as we see we are continuously moving forward and our children are safe in their schools Schools are just do you want to add anything to that? Yeah, no just that I would echo what you said, mr. Mayor, you know, I've heard that on on on other occasions as well I don't want people to be very clear the decision to go remote is always the last option Uh, our kids need to be in school. This is where they also get their healthy meals As well This is what allows parents to be able to continue their day to get to work and do the other things that need to happen Whenever we make a decision to actually close schools. It is a major major disruption Um, and so it is it is really only used really as a as a last resort Um, and and I think in this case while this was a uh a tough day in terms of the rain Uh, it certainly did not put our our kids are not in danger All of our schools are open our teachers are in school our kids are in school and uh, and and doing and doing well We do have some schools which have had some flooding and we have to address that But but we treat taking the option of the remote learning day very very seriously And we use that as a last option It's just really commendable on how professional our school system our first responders On the exact exact team during these issues of Whether emergencies we still have to operate a city And our ability to do so is really commendable with the coordination from this team The city still has to operate with minimum disruption And that's what you saw in our school system as well as as of the deputy mayor pointed out That you know able to go in and do those rescues when it's needed and to notify people The city must continue to operate and we want to be clear on clear on that Zach can you talk about uh, the state of emergency state of emergency? Uh, that we're calling is based on conditions Um, not not based on predictions, you know, it's based on what condition but Zach can you go through that please? Absolutely, uh, so there's a number of different uses for a state of emergency and and I appreciate the question Um, you know the the issuing a state of emergency at 8 a.m. 7 a.m. 9 a.m. Or 10 a.m It's it's not going to impact our response or the work that we're currently doing A lot of the state of emergency is more about Things that we can then put in place should we need to around contracting around different rules Things like if we were going to issue a travel ban, which we are not at the state right now Where we would actually implement a travel ban, you know shutting down roads not enabling people to freely move around the city Uh, it's also something that becomes very very important as we move into the recovery phase Right, one of the things that's needed is is as we start to assess the damage that has occurred to either people's homes private personal property Businesses the state of emergency is something we can then use as we are going to the state and FEMA For additional support to aid and individual assistance Business loans or grants that can help people recover After the flood waters have receded I would also add that The governor issued her state of emergency This morning around the same time the decision was made here at city hall Mayor Adams made the decision around the same time So just because we're having the briefing now around 11 30 or 12 whatever it is right now Doesn't mean it wasn't uh the decision wasn't made earlier today Next we'll go to mark morales from cnn mark Mark you should be able to speak now All right, we'll go to the next one next. We'll go to craig mccarthy from the new york post craig. You can unmute your line Mr. Mayor, I just want to ask um So you say you don't need it your administration is saying you don't need to issue a state of emergency at 8 a.m Or 7 a.m. This is about recovery But when you're in the national weather service issued an alert at 8 24 this morning calling the flash flooding warning life threatening So why did it take hours then for you to tell the new yorkers to stay home? Uh, uh, zack, can you respond to that please? Yeah, I just want to issue a point of clarification What I went through is a whole list of reasons that you would use a state of emergency Preemptively you would do it if the situation might call for that in this case There was no need for additional resources that a state of emergency might provide Or things like, you know issuing a travel ban. There was no need for a state of emergency earlier to be able to do those types of things I have been in very close contact with my counterparts at both fema And the state around whether or not there were additional resources We might need to request, you know for things like life safety events But we have the resources here in the city through our incredible partners at fdny Ems nypd to be able to respond to those types of situations. Um, so I hope that answers your question Next we'll go to emma fit Simmons from the new york times emma you can unmute your line Hi, can you hear me? Yep, we can hear you emma Hi, um, good afternoon. So, um Mayor, I'm curious, you know, why it took so long for you to speak to the public Governor huckle was on the radio last night mta officials held a news conference and A lot of new yorkers, you know, we're we're worried this morning And facing these implications and why didn't they hear from you sooner? You know, I think it's so important as I've stated over and over again this administration operates as a team And I want my commissioners my deputy commissioners the leaders of this team Who are closest to the ground of a situation to communicate and that's what commissioners school did There was not an absence of a voice of this administration our team leaders That are on the front line. I'm you know, I'm just really pleased that I have strong competent leaders That understand their roles that they're supposed to play and I give them a clear mandate That if something is dealing with a particular issue in your catchment area I want you to lead from the front and you see that over and over again You see it with the chancellor you see it with of commissioner cabernoff You see it with all of our leaders and that's what uh, zack did you heard from my representative And he did a good job of informing the public and that's what you're going to continue to see Throughout this administration. We have good team leaders that are competent that understand the subject matter And they know how to lead The leadership is not only the mayor it is all of those who are placed in those positions and that's what you saw And just to go back to craig's previous question. He asked about why Timing on a travel advisory So I would just again point out a travel advisory was one of the first measures put in place by this administration And the notify nyc alert first came out at 250 yesterday. So the advisory was in place That's a different thing than a state of emergency The Commissioner iskall wants to add something commissioner. Yeah, so he's just confirming. Yes the travel advisory went out yesterday. That's correct All right, we have time for two more We'll go to annie mcdenna. Annie you can unmute your line Questions, uh, first, do you have advice on where people living in basement apartments or experiencing flooding should go Where they should move to higher ground in that event or their city locations where they could go and access Somewhere to be if they're experiencing flooding at home and second for the mayor Can you talk more about what you saw in east new york this morning? Were you examining street conditions talking to residents? What exactly were you doing there? the the issues around Basement apartments and where we're going to have people the first order business to make sure we properly rescue And we will have sites for those who can't go with families and friends We will always provide shelter for those who are in need and we will navigate that I was in east new york over at the christin cultural center for unfortunately A loss of a police a former retired police officer And while I was there, I took the opportunity to go through east new york and to khanasi Into the flat bush area as well as the sheepshead bay area to get an observation On what was happening on the ground prior to doing this press advisory Our last question will go to joe anuda from politico joe. You can unmute your line Joe going once Hey, mr. Mayor I was wondering actually for the oem commissioner commissioner iskall Can you just give us a sense of the media hits that you've mentioned that you did yesterday to warn new yorkers? And then for a dp commissioner, can you talk a little bit about When you started clearing catch basins, how many you cleared just describe the operation a little more. Thank you Yep, absolutely. I did a 1010 winds hit and uh, and then did a fox weather hit And then we've also as the mayor has noted sent out the notify nycs the press releases the travel advisories social media posts in addition to all the other messaging that was coming from the city All right, that's all the time we have and then hold on. Do you want me to answer the other part? Oh, sorry about that commissioner Yeah, um, so just uh Yesterday morning the flash flood. I don't have the precise time. Uh, the flash flood plan was uh, activated yesterday morning And as per that plan dep along with several other city agencies sanitation dot a couple of other agencies that have crews on the roads We have a list of priority Intersections which are highly vulnerable to flooding or are important for some way in fact over the last two years We've worked closely with the mta to make sure that catch basins that could affect subway stations are on that list I don't have an exact number. I would probably get it Later today of how many catch basins we checked and cleaned in those 20 from in those 12 hours I guess before the rain started but that is what we do the day before As well as get ready for uh for the staffing and and that's why We actually canceled some holidays that people had scheduled and things like that to make sure we had full crews on site Today across the city Thank you commissioner. Thank you everybody for joining us today. If you have additional questions, please reach out to our press office Have a good day. Be safe. Thank you


G'day and welcome to this demonstration of the Moodle mobile app for iPhone Okay, I'm presuming you have an iPhone and you found your way to the app store and downloaded the free app It's called my Moodle So we've launched it now the first thing the app will do is ask us for the site URL Okay, that's the address of your Moodle site So you enter that and you enter your username and password Okay, so that's been done. We could in fact revisit that and you can manage multiple profiles or if you like use a logins to a said Moodle site or multiple sites for that matter and Then toggle between them. Okay, so they've been added. Let's say we in fact want to use the student profile it returns us to the Main page for the app. I guess it's the dashboard or the home home screen We've got four options upload Participants web and help so let's start with upload. This is the good bit This is where we get to share or contribute or upload media From our phone to our live Moodle site So the first option there is to browse photo albums You can navigate your phone through to your albums We've got a photo here in our Camera roll We get a preview and then we can select upload So that's getting sent from the iPhone to your live Moodle site and in fact this uploaded media will reside in Your private files block You can decide if and when it's appropriate to share those files with others. It might be It might be media attached to a forum post It could be a file that needs to be submitted as part of an assignment or you can upload that the media for a lot of other purposes In Moodle. All right, let's look at the next option there within the upload menu and that's take a picture or video So we're now going to take a picture like so so we get a preview of The image there on our phone we can either retake or Use the preview that's been presented. So we'll select use Okay, so the dialogue there indicates the uploads being completed again that that image will find its way to our private files block Now live Moodle site this time. Let's revisit the same function, but we will not take a picture We're going to take a video. So we need to toggle To the video menu we click the red button To start recording and when we're done we click the red button again to stop So the dialogue here is we can either retake the video we could play it or we can use it So let's select use Again, the media is going to be uploaded to the private files area of our live Moodle site The last option there is to record audio. So we hit the green record button And we start speaking we are speaking live as we are Demonstrating the Moodle mobile app for iPhone and we'll stop the recording there That's been saved. We could Play back The recorded audio by the replay option or else we can send that directly to our Moodle site And it actually tells us the size of this file Upload so we'll navigate out of the upload area The next function there on the dashboard is participants So we select that we get a snapshot of the courses We're enrolled in as a user under under the profile. We've selected currently We can't directly access the courses or the activities resources in them, but what we can do If we select one of these courses, we we're presented with a list of course participants and via our phone we're able to view the user the other course participants profiles and Communicate with them directly So we're looking here at a Fellow course participants profile. We can see their name their avatar inscription about them We could contact them directly via email using our phone if we've got Mal configured for our iPhone if the said course participant had a Phone number entered into their user profile. We could with a single click of a button call them direct via this page You can see their location We could send them an instant message This is a Moodle instant message by the way not an SMS The recipient will see that next time they log into Moodle if they're offline It will in fact get delivered to the mailbox We could add a note to this user's profile That would be on the on the presumption that we had the the right capabilities and permissions to do so you would likely need to Need to be an administrator or a teacher Or you can add This user as a contact And they get added to the address book on your iPhone never get out of The participants function the web function Will display your Moodle site on your phone And give you access to the full functionality. So you may need to zoom a little bit there But the idea will be you will be able to log in access your courses And view activities and resources and so forth Bearing mind that the the the Moodle mobile app isn't meant to replace the the core functionality of of You know the Moodle software and how you might access it from a desktop purely add value to it particularly through adding contacts communicating with them via your phone and Uploading media to your private files area The help function gives you direct access to Moodles web-based help documentation the task queue function is Very handy in the sense that you can use Moodles mobile app while you're offline To capture media for example or communicate with fellow course participants and You can queue those tasks to be sent when you your phone next connects to the internet and as we saw We can toggle between various user profiles that we've entered into the application So that's just about it. I hope you've enjoyed this demonstration relax enjoy and happy moodling


Yeah, so this week, Bob is going to check us through Glorut and Monin. We're going to see what we can get out of the two monitoring tools. And yeah, it's going to be quite interesting, particularly interested in Glorut. So this is the third time we're having subadmin meetings this year. And yeah, so Bob. Thanks Tito. Good morning everybody. Let me see if I can share my screen. Okay, yeah, good morning. Everybody see the screen alright. We should be seeing a browser window with Glorut.org. Yes, we can. Yeah, I can see it. Yeah, okay, so I don't know. Today I spent a little bit of time just going through this useful little tool that we found has solved many problems over the last few years. I discovered it about four years ago, four or five years ago. We used it to diagnose some issues that they were having with a measles immunization campaign in Bangladesh.  It's a big and complicated piece of software. Everybody has different kinds of user loads, different kinds of sizes of databases, different types of programs that they're running. And so sometimes it can be hard to know what's going on when things are slow or when things are crashing. And so it's really important to have some good monitoring and diagnostic tools on. And certainly one of the most useful ones I found is this Glorut. Glorut is particularly, it's a JVM agent really. It's essentially looking at what's going on inside your Java virtual machine.  But it's only one part of the picture, of course. I mean, as you know, the DHS2 in order to make it run, there are other pieces involved as well. Importantly, the database and also the web proxy and also the operating system that it runs upon. If you've been using the DHS tools that I know some of you have for some years, even if not some people have set up independently, you'll know that it's also important to monitor other aspects of the system, other than the JVM. And by default, we installed this thing called Moonin, I've got it running here. And thanks to Kwame and Oswald, we're actually looking at the live system in Ghana. Just because looking at these monitoring tools on test databases, I won't really tell you much. When you have some live data, it's important to look at. The thing about Moonin is that by default, you have hundreds and hundreds of graphs. Probably too many. That's a big problem with a lot of monitoring tools. You have lots and lots of information. It's important to know which ones are useful. When I am looking at Moonin, normally the kind of things typically I would look at and put in this overview, go to the host machine and look at the system. This is one of the most one-star points and you have a look at what your CPU usage is like. If you've got a lot of excess CPU usage, then you know you've got some kind of problem to investigate. There's a little bit of a problem here, all right, that I hadn't really figured out before. This is usually, you see on the CPU, this is kind of normal use down the bottom. There's this purple stuff up above. The purple stuff is IO weight. I've not had the opportunity to investigate this, but Oswald and Kwame, this is something to make a note of. The CPU is spending a lot of time in IO weight. I don't know if anybody can give me a suggestion why they think that is. Why would a CPU be spending so much time in IO weight? What does that indicate? So, I think I had that issue before from Serlion. It's because the RAM is old. If you have a very old, some of these hosting provider, what they do is that they give you a specification and then they don't tell you the type of RAM. And if the RAM is very, very old, it spends a lot of time on IO weights and then you have these spikes and then you don't know why it occurs. But then when we requested to them based on the specification of the RAM, we send a request to them that we want them to change the RAM. That was fixed. I think we had that problem for a couple of times. I think we contacted you and you yourself was not surprised at it. But that's the main reason why I wrote. But you mean the disk? With IO, you're typically talking about the disk. I think it's the technology. It's better to use the SSD disk rather than that disk. I think it can improve the thing. IO weights generally, what it means is that the CPU is waiting for an IO operation to complete. And that's usually a disk operation. This server generally has been, and I know, Gerald, you had that problem with DaddyServe that they gave you very bad disks. This server generally has been performing reasonably well. I'm not sure why you have this here, but it's something that we probably should investigate. But anyway, this is one of the first things I look at, right? Look at the CPU usage. We're not here to solve problems today. We're just looking at the diagnostic tools. So yeah, this is something to look at. And Oswald Kwame, I think we should make a note. Let's see what's going on with that disk. We don't expect to see that. This is running on LinNode. LinNode disks are usually very good. So it might be something happening. Other thing that's in to look at, yeah, the memory usage. Memory usage here, what we can see is, and this is on the host system. This is a 64 gig host. It's got a lot of this purple stuff on the top. This was a reboot that happened last night. I'll talk a little bit about the reboot later. And so this yellow stuff is actually unused RAM. You might think you need to have a lot of unused RAM, but that's not necessarily the case. I mean, you pay good money for RAM, so you do want to use it. You just want to make sure that you don't run out. Now, what the operating system will do when it's got free RAM to spare, this particularly the database, it's going to make use of that as cash. So seeing all of this RAM being used as cash, it's not a bad thing to see. That's a good thing to see. We can see that's been fairly healthy over a period of time. This is like over the past week. So RAM seems to be quite well allocated on this machine. There's quite a lot being used. And whatever's left over is being used as this cash, which is a good thing. We don't like to see too much swapping. There is a bit of swap happening. This is this red stuff on the top. Something else I think we want to investigate because too much swapping is excessive use of the disk. It's generally not good for a database. I don't want to spend too much time on Moonean. That's usually why I start. I look at that. The other thing that's very useful to look at on Moonean is your database, your Postgres. Moonean database plugin. We should look at the disk first as well. Let's just go up here and look at the disk. One of the things that generally seems to determine the difference between a well-performing DHS2 system and a well-performing database and one that isn't performing so well is disk latency. And so I always have a look at this disk latency here. The disk latency you can see generally is OK. This SDB is getting high 27 milliseconds. Generally, these kind of figures here blow 10 milliseconds or into the microseconds is more what you like to see. What I can see here is generally those figures are good, except for a little bit of extra activity that seems to happen in this dev SDB. The dev SDB, as far as I recall on this system, is the swap disk. So it could be that some of these high latency figures and some of that swap memory that we're seeing used from the other graph is related to some issue with the swap. So again, this points to something that probably should be investigated to see whether we can improve some performance. It's not good to see maximum latency figures like this. You've got high latency. This is when your CPU is going to be an IO weight. And you always want to see your latency figures somewhere less than 10 milliseconds. For the main disk of SDA, that's where the database is sitting on, latency figures are very good. It's never over 1 millisecond. So the actual access to the disk and stuff is good. There's something going on with the swapping. That's maybe not something we need to look into. But disk usage obviously is useful to keep an eye on. And you can see here with a disk usage that we're currently using 50%. You see this little spike that happens. You expect that that happens in the middle of the night when the analytics is running. I think we look at it over the month. I mean, look at it over the year. It's funny we haven't been seeing this. Usually expected to see this spike happening every day. We don't need to worry about running out of disk space at the moment. We've seen a kind of growth happening here. Something happened back in October. We don't know what it is. Maybe Oswald and Kwame will know. But for some reason, our disk usage, it must have cleared out a lot of files all of a sudden for some reason. I don't know why that was. Yeah, I think back in October, a station in town came up on the same machine. One of the invasions before we do an update. I think that is part of it. Okay, so whatever happened back in October? It was almost the update. The backup spread to take off some backup that was installed on that day. Okay. Anyway, whatever you did back in October was a good thing. Because now it looks like your excuse, which is quite stable. It's growing a little bit each week, but it's not going crazy. The other thing I look at on Moonin is the Postgres plugin. And probably the most important graph to look at on this one is the connections. Again, this is reasonably healthy. When you see 80 like this, approximately 80 connections, I think there are two databases here. One is a staging one. Okay, let's look at it on here. There's the DIMMS database and then this is a staging server here. It's averaging around 80 connections. What that typically means is that the disk, the pool, the database pool, and we're going to look at this in a bit, probably just set as a default value. Interesting. When you look at the comment in the DHS2.com file where you set the maximum connections, it says the default is 40. But I think that documentation needs to be updated because the default is actually 80, not 40. So you can see what actually was the case here was that the number of database connections was set as at the default. It hadn't been set to anything higher. One of the things we did last night, because I thought it was maybe a bit low, is we increased that to 200. I'll tell you a little bit about why we did that. When you look at the GlowRoot. But this generally is a useful graph to look at. Kind of things to look out for here. You see there's a bit of time with some of these connections. They're waiting for the lock. You kind of expect that if you see too much of it, then you know you're getting into some kind of problem that connections are deadlocking or blocking one another. But generally this pattern seems to be fairly consistent. It's important to realize, I guess, that the way Moonean works, Moonean is a fairly simple tool. It just takes a sample every five minutes. So you're getting a kind of average picture of a snapshot of how things look every five minutes. That gives you quite a lot of information. But also it means that you're not getting the kind of instantaneous information what's happening in between those five-minute intervals. And I think what is happening here, even though you don't see it, is that there's spikes happening here where sometimes you actually look for quite a lot of connections in a short time. That's simply something to do with the way the DHS2 works. I've increasingly come to understand that a lot of the API calls, it may be a single API call, but it can sometimes have tens or even hundreds of database queries that happen as a result of a single API call. And that can mean quite a sudden rush of database connections required, which wouldn't necessarily show on here. Anyway, we'll look at that more when we go to the code. The other plugins on here, okay, the actual plugin, the Nginx plugin is kind of similar. This kind of thing is it just give you a good idea of what your daily load is like. Your access is per seconds, typically peaking at around 50. And is that more or less always useful? I find these kind of long-term graphs quite useful. I understand, you know, how has the system been changing over time? Not hugely. It's got to been a little bit busier in 2023 than it was in 2022. And the load is, yeah, getting up to, it was about 70 per second earlier this month. I guess that's probably related to, because this is an aggregate server. This must have been the week when most of the reporting was happening. That's when you expect to see a higher load. There's a plugin that's missing on here that hasn't been installed, which is the Tomcat plugin. As well, we need to talk about this. We haven't enabled the Tomcat plugin on Moonin. Tomcat plugin has some quite useful information to look at as well. Unfortunately, we won't be able to look at it here now because it's not plugged in. Okay, so that's your kind of very broad overview of what's going on, as Moonin gives you a reasonable picture of whether your system is functioning healthily. And I suppose, yeah, my big advice when looking at Moonin is, you know, you have these handful of charts that you know is useful, amongst quite a lot that are less useful. And it's always good to look at them. If you see something strange happening, then also look at, well, what's been happening over a longer period of time? Are you seeing anything drastically changed? So, for example, I went to that disk. We saw this problem. We were seeing this problem with swapping onto disk. So I, wait, is this something new or has it been happening forever? It looks like this is a problem with, as always been there. It's not something that's recently developed. Something that, yeah, probably should have been investigated back here as to why it was happening. It doesn't seem to be causing huge problems at the moment. There's quite a lot of excess CPU capacity on here, as you can see. All right. That's Moonin. But the main thing I really wanted to talk about was this Glowroot. If you're looking for the source of Glowroot, this way comes from glowroot.org. You can read a little bit about it. What it says is that having it on your system causes very, very little overhead. And our experience has been that is certainly the case. There's no great performance problem with putting the agent on. And as we will see, it gives you lots of information. How you actually do it? Well, again, if you're using the kind of automated installation tools, it'll do it itself. But I can show you here, I think. Let's just have a quick look. It's installed on the Tomcat container itself. And you'll find it at the place where all your Java Office and things are set. The place where you set your Java Office with the standards Ubuntu or Debian installed Tomcat is in this file, et cetera, default Tomcat.9. And in here, we can see that... I haven't looked at this file before. Okay, I see it now. You've got 18 gigabytes of RAM allocated to that. JVM seems to be about a good amount. We'll look at a little bit of that in a bit.  And that'll start up this Glowroot thing. It has a web-based interface. It listens on ports 4,000. I should be able to see that. Here it is. There's your Tomcat running there and this 4,000. You see it's the same PID. It's running. It's the same Tomcat. It's listening on a different port. That's your Glowroot profiler. It does mean that if you want to look at these Glowroot charts and things, you do need to enable a location on your proxy server to access that port. So that should be opened on the file. Yeah, we're allowing proxy server to access 4,000 on port 8080. So that's simply so that we're able to look at a Glowroot. Okay, so let's have a look at it. This is kind of the main graph of the page you come to when you log into GlowrootJava. I had it showing you the web transactions. And you can change the time period here for the last four hours, the last half an hour, to the last 30 days. Again, like with Moonee, it's good to see how we're looking today compared to the way things are generally in the month. And yeah, you can see that things were a bit quiet and then they got a bit busier, then they got busier and busier still. And this week is starting to get quieter again. And I suppose this is because we've come to the end of the reporting period. General load on here. Yeah, that throughput is getting up to about 3,500 minutes. Busy enough server. Ghana is a big country. I think their DIM system is quite big. On the left hand side is a list of all the API end events, ordered in terms of how much load they're putting on the server. So one of the things we can see from here over the last 30 days is that 61% of the load is related to these analytic calls. One of the things that you can conclude from that, often if you're trying to just generally improve the performance, there's a law to this. It's called Amdahl's law, which says if you're trying to improve performance, essentially you want to try to find the thing that's consuming most of the time. And if you make that a little bit better, your whole system is going to be better. If you find something that's used very rarely, a very small part of the system, but it's actually very inefficient, and you improve that, then the impact on the whole system is going to be much less. So it's useful to see requests ordered like this, because if you were trying to do a general improvement on things, you would start here and say, well, this is analytics, obviously, is using quite a lot of CPU. Anything we can do to improve that we should, followed by this custom data set reports. It's much less than analytics, but it would be the next target to look at. So as you can see from global, it gives you a lot of information, just looking at this front page. Looking at it, not in terms of throughput, but in terms of its response time, I think this is actually the front page. You can see that the response times are generally quite flat and quite low, and every now and again we get quite big peaks, quite big peaks happening. What we'd like to do, obviously, is to reduce those peaks, right? And so we have less. As I said, I did a little bit of a tweak on here last night in the hope that we'll actually reduce a bit. Things haven't been as bad the last couple of days as they were the week before. Last week, obviously, was a little bit stressful. We can have a quick look at what was going on last week. One of the things that's a little bit interesting looking at this is that, again, a lot of this time for these long-running requests, they're not actually doing anything, they're waiting. And then quite a lot of JDBC collection going on. Usually when you're seeing very slow response times, what you expect to see is that there are very heavy JDBC queries going on. But as you can see here, the contribution of those JDBC queries is not so much. So if there's any slowness in this system, it's not really about slow queries. Whatever's causing it to be occasionally a bit slow is related to something else. And it seemed to me that this waiting that it's doing and this time it's taking, getting connections is potentially a bit of a problem. Let's go back and see what's been happening today. This is, I don't know, what time is it in Ghana now? I think we're just about over the hump, a little bit of throughput again. We looked like we hit our peak time about an hour ago. Does that sound about right, Oswald? And now we're still busy. It's kind of over the hump. It's about 12 midday in Ghana now. Okay. So it looks like things are slowing down a bit. We're kind of just about over the hump. I think that's fairly difficult. If I look at it over two days, yeah, a very rapid, very rapid leap of activity in the morning and then things by 12 o'clock or 12 o'clock my time things start to ease up again. So the traffic pattern today is very similar to the way the traffic pattern was yesterday. How has the performance been? Well, let's look at the response times today and yesterday. We had a nasty little spike yesterday around about there. Then another nasty little spike a little bit later on in the afternoon. We haven't seen any of that yet. So it's a good thing. We'll know a little bit better towards the end of the day, how it looks. If you do have, okay, the other thing that's, if you want to get into more looking at, looking at this in general picture, if you like. The other thing that's worth looking at besides just the web transactions, I'm going to come back to the web transactions in a bit, is to look at the JVM itself. And by default, it's showing you this, this graph with the deep memory usage. And you can see this deep memory usage is kind of bouncing around, around nine gigabytes, picking up to about 11 or 12. We know this machine has got 18 gigabytes allocated to it. So this is a very healthy place to be. There's no problem with the RAM allocation. If you do have problems with, with basically not having allocated enough RAM, then often the symptom that you see is actually excess CPU usage. It's kind of a little bit counterintuitive to expect if you see too much CPU being used that you need more CPU. But often that's an indication that you're struggling with RAM. The reason for that being that when the, when the JVM starts running out of memory, it has to start getting much more active with his garbage collector.  And sometimes you find then that 90% of the CPU is just being used up by, by trying to keep reclaiming memory. Those kind of problems are quite easy to diagnose with blow root.  These two graphs, the old generation and the young generation, you would see, this thing is peaking at 5.8 times per seconds. That's not, that's not, this is milliseconds. So it's being used like 5.8 milliseconds per second. That's not catastrophic at all. This is just a garbage generator doing its normal business. And the old generation garbage collector is hardly being used at all. So this is an example of a system where there is no real issues to be concerned about a memory allocation. You might even argue, perhaps it's got too much. If you needed to get that RAM and allocated to something else, maybe give us, maybe you could even give a little bit more to the database server, a little bit less to the Tomcat and it would still be running all right. You can look at the actual CPU usage for these graphs as well. Again, we're not going to see anything very interesting. You can see it's the green and the purple, generally very low CPU usage. So yeah, the JVM graphs, these ones, generally they're very useful to look at when, when you've got problems with CPU and RAM. This would give you an indication that, you know, the things are not properly configured. In this case, fortunately or unfortunately, it depends which way we look it up. We've got no real traumatic things to look at here. Everything seems to be looking good. So let's go back to the transactions. I'm still half hoping that something bad will happen, but I'm also very happy that nothing bad is happening. What I was suspecting is that because the connection pool hadn't been changed from its defaults, and because this is quite a busy server, I don't know, you've got like, there was about 8,000 users the last time I looked, maybe it's more, maybe it's less now, but it's quite a lot of users we saw from the throughput that it's got, it is with fairly heavy traffic. It probably needed to have a higher number of connections than what we were giving it with that default pool size of 80. I've changed and the fact that it's spending quite a lot of time waiting, the fact that we're seeing little to time where it's taking a lot of time getting the connection that's sometimes an indication that the pool is not configured quite how it should be. To show you the changes we made to the pool last night. Let's first of all just look at the diagnostic. The other thing that's worth looking at is the slow traces. Now slow traces can be a bit slow to open, if there's too many of them. So I find it sometimes is useful. Let's look at a smaller time period. We're just looking over the last seven hours and let's look at what's been slow in the last seven hours. How we can see right through the night everything is fairly quiet. There's a few things here taking quite a long time. That took five minutes, this took five minutes. This must be something related to analytics. What are these things? It doesn't get you into the detail of it. These are some very big analytics requests. I'm not sure the detail of what it's about. It's a heavy one. It's taking five minutes to run. And this one, the reason for the long time it's actually just the query itself. So this thing is doing quite a heavy query. I hope that this is not a query that's going to happen too many times. If you suspect that something you can do to improve the query, you can look here at the actual queries itself. And you can see this is the culprit here is this one. This query here. For some reason it's slow. It looks like it's going over a number of years. The yearly from an Olympics. 2020 and 2019 and 2018. If we reasonably this thing was causing us a problem, then it would be a case go to the database, perhaps taking this query as it is and running it through, explain and analyze and find out exactly what's slow about it. And maybe seeing if there's ways it could be optimized. I can't give you a diagnosis here and now. Looking at it, I would have to, as I put it through the analyze and see. Again, you can make a note. It might be worth. I don't know how often this happens, whether it's just a one off or. Happens that are interesting time very early in the morning. Somebody is running this. This thing here is similar amount of time. I would like to be the same query. Obviously. It's an analytics query. Somebody different. And it's taking quite a long time to run. So we've got a pickle enough server, I guess where it's got some analytics requests, which are very heavy. And I've got one suggestion. I made one suggestion last night about something to do to perhaps avoid the problem. If it happens a lot. A useful thing. If you do have a troublesome query and you want to raise it with developers, I think one of the most useful things about Glowroot is that it's a good. It's a good way of communicating. If you have problems instead of, you know, going to go back to the developer saying, no, my system is very slow. You can tell them very specifically. I've got this query here that is very slow. Is there anything that you can do about it?  We can take one of these things here. And say, well, this query is blurring me. Is very slow. Can you analyze it? If you click on this thing here, download the trace. The reason you can download this whole queue as a zip file there. And then you can send that trace file for somebody else to look at. This is much better than taking us. People often take screenshots. Right. They take a screenshot of this and show it to me and say, Bob, what's going on? It's much easier if they just download the zip file and send it to me. Because then I can go into the query, start some things and see what's going on. Okay. One of the suspicions I had. And I'm going to look at not so much. This is a classic one here that's often very slow. Now this gets metadata action. But typically this takes about five seconds. One of the reasons I pick on this one is because when you look at the, when you look at the request, one of the things you note is that it's only one request, but it results in 300 queries to the database. So a lot of little queries going on as a result of one request. So you can imagine if this comes, or if two or three of these come up at the same time, then suddenly there's going to be a lot of attempts to gain, to gain database connections and to execute them. So it's not so much that particular queries are slow. If I look at this, well, in fact, one query is quite slow. I've seen this before, selected element being slow. That's something to kind of maybe look at. But the fact is also that there are many, many, many, many different queries. And sometimes if you have a flurry of queries like that, you may find that your database pool kind of instantaneously becomes a bit clogged up. And that's the reason why I made a few changes. I'm going to show you the changes that I made in the comp file last night with Oswald. And we just have a quick look again to see whether it's made, had much impact. And then we open up for questions. So, yeah, if you've got DHS, your DHS comp file is in here. I'm going to try to avoid database password, but particularly as it's being recorded, otherwise we'll have to, this is good, I'm looking at the file from the bottom. One of the things that wasn't figured before was this server-side cache. This is something that I'd strongly advise you should always enable, right? The server-side cache. Because basically every time you make an analytics query, if you cache that at the server-side, then the next time somebody makes the same query. And this is often the case if you go and you open a dashboard and then you log out. If you go back in again, you give back to the same dashboard, you're going to be making the same query again. So dashboards do cause the same queries to be executed many, many times. It's a really good idea to cache them like this. That reduces the strain on the database a bit. Oswald, I don't know. I don't have access to the front-end. We were talking last night. Did you find that your dashboards are loading a little bit this morning than yesterday? Yes. I can't take this one. I'm doing a lot more. Is it kind of noticeably different? Yeah. Because it should be if you enable the cache. Because it means that those dashboards should be coming now without having to always go back to database to get the data. So that's one change we made. And definitely this is an aggregate system. I know a lot of people are more interested in tracker performance problems, but today we've been looking at an aggregate system. This thing is, it can make a very big difference. As we saw on the Glowroot graphs, most of traffic on the server, most of the heavy traffic on the server is analytics related. So anything we can do to improve that. So the other change that they've made on here, just up here somewhere. Yeah, here. The pool size wasn't set, right? It was just commented out. You see the comment says default 40. It's actually wrong. The default is actually 80. I checked that the maximum connections that's been configured on the database is 400. So we can easily handle more than 80 connections. So we increased that from 80 to 200. And these two settings here, this is something I discovered some months back. I think it was looking at your system in Rwanda. We saw mostly where you had an external database that one of the things that the pool does, it maintains these pool of connections. And when it takes out a new connection, it'll test it to see whether the connection is good. And the problem is when you have a request like the one I showed you there, it's going to have 300 different queries executed off it. When you have a request like that, and it maybe suddenly requires a lot of connections, then the process of acquiring new collections can be quite heavy. So one of the things that you're able to do is to say, well, don't test the connection when you check it out. Just check it out and try it. It works, it works, it doesn't, it doesn't. But when you check it back in, in other words, when you make it available for some other transaction to use it, instead of that point instead. So we're not putting bad connections back into the pool. So those are the three changes related database pool that we tried out from last night. And we want to have a look through the course of today. So far from this morning, everything looks okay. It's not dramatically different from yesterday, except it does seem that dashboards seem to be loading a little bit faster. And we haven't seen any spikes yet, but it's too early to say whether we are going to or not. I need to keep an eye on this. Let's go back to this thing out of the way. I mean, we're actually over the big peak at this time. If we look at the response times over the course of day, eight hour day, everything is pretty good, right? Averaging 100 milliseconds, no big spikes. The slow traces that we're seeing. The other thing to worth looking at, in fact, is the percentiles. Centiles just shows you 50% of the connections are down here. So 50% of the connections, I mean, they're giving you a response in 7.9 milliseconds. Up to 95% of connections are giving you a response time of 200 milliseconds, which is not bad. And we're having very few, and we saw some example there. So there are a few quite heavy, but they're only 1% in the 99th percentile where we're seeing a couple of these going up quite a bit higher. We can see how does that happen compared to yesterday. Let's go back two days. And, yeah, so far it looks like we managed to maintain the flat, even on those heavy ones. But as you can see from here, if you were looking yesterday morning, it wasn't actually looking that different either. But you're interested to see when we look at this again in the afternoon, have we got rid of some of those? Okay, that's enough for me. We can talk about Glowroot and looking at different kinds of problems with it all day long, but I want to stop there and open up for any questions. And thanks to the Ghana Health Service team, this piece of Ghana in waiting for allowing us to use their server as an example. If anybody else wants to volunteer theirs, in a subsequent week I'm very happy to do a performance review of. So particularly if you have something which is creating particular problems, we can look at it together and not give away any data necessarily. All right, any questions? Hi Bob, this is Mahindra and recently I have joined his Pindya. So actually my question is that I have set up one server and there is no any Alexi container. And I have set up DHS2 application as per the document. And I have also set up Glowroot over there as the instructions. But as this is a blank database, there is no any data right now, but I'm not able to see any kind of like there is API slash 37 or analytics, like the data volumes, data elements, these kind of logs I'm not able to see over there. Well, if you have, I mean, if you just set up a blank database. Yes, yes, yes. Then, then, then, yeah, you're not going to be seeing any traffic. Okay, okay, okay. That's why for this demo today, I had to ask permission to use, to look at the Ghana one, because if I just set it up on my laptop or something, it works fine. But then there's nothing really to see. So yeah, what I'd suggest you, what you should do is put Glowroot on one of those production servers. Okay, fine, fine. Thanks. If it's one that you, that you know is having some trouble, then you can share what you see. A question, a question. Yeah. Well, thank you for your presentation. And now I have to question questions. First is, can we set these environments in a way that they can be centralized? What I mean, let's suppose environment where you have multiple servers, and maybe the system administrator want to get all the information from multiple server in just one server. Is this possible? Secondly, what is your take on the difference between this and Prometheus and Grafana? Thank you. Okay, two good questions. First of all, I'm not sure who will take first. Yeah, Glowroot can be set up in a centralized way. You need to look at the documentation. I'll show it here somewhere, but you can have a centralized collector for Glowroot. Have it, you know, have everything centralized to a Glowroot panel. It's not something I've done often because first of all, it only makes sense if you are managing quite a lot of systems. I know in Rwanda you are now managing quite a lot. And, you know, it tends to be particularly JVM related. All of these metrics, in fact, are just coming off JMX. On the question of Prometheus and Grafana, what I will say is that they're both very good looking. They're both very good looking environments, particularly Grafana gives you very nice looking dashboards. Quite a lot of configuration involved. In order to package something out of the box to give you good value, it's quite hard to do in a general way. I think if you're setting it up in your own environment and you have time to tinker on it and customize it, I think it is possible to do it quite... You can get all of these same metrics, for example. All the metrics that you're getting here off Glowroot, they are just coming out of JMX and you can query your JMX with Prometheus. But it would take you a lot of effort and a lot of work to put together views like what you're getting off this one. So, yeah, my view of those monitoring... I like them a lot, but it takes a lot of effort to configure your environment properly. One of the things I know that... I mean, one of the reasons that we have Moonin running, for example, is not because Moonin is very beautiful. Moonin is quite horrible, I think. But it's really easy to set up. And so almost for free, you get lots of useful information. One of the things that I know Tito has been looking at is also Xabix. I quite like Xabix. It's much better looking than Moonin. There's a little bit more configuration work required to get a good environment going. Prometheus, I think if you've got a very docker-centric environment particularly, you have everything running on docker, then Prometheus works quite well and it doesn't require docker, obviously, it fits into that paradigm quite well. We have some metrics exposed on DHS2 using Prometheus. I've had a look at some of them a few years back. I don't think there's anything particularly there that's gonna give you more information than you're getting off this and also the proxy log file. So I haven't put too much effort into it yet, but I'm very happy to look at anything that people might have done in the community. If you've set up a nice Prometheus Grafana environment, the other one which is very popular is the Elk Stack. The Elastic Search and Longly and Kibana. I think that Danny and the Solid Lions people are very fond of the Elk Stack. Manzi, that's it. I think they're very nice. They're quite hard to package in a way that they're all gonna come pre-configured out of the box. Thank you. Let's see who is on here. Okay Bob, I've got a couple of queries. If any of these systems have a notification or alerts system to send out, if anything has been recorded out of the ordinary? Yeah. So who's this? Sorry, it's Damian from MSF. Hi there. Yeah, alerts is a really important thing. I don't know if, actually not checked, whether you can do alerting of glow. It's more of a diagnostic tool and a monitoring. Mooding for sure. If you look at Mooding, you see there are, you can set thresholds here on all of these matrix. So you can see there's six critical. Something has, has triggered something over the threshold here. I don't know what it was. Yeah, that's a, that's a skew stitch on this thing, which is a virtual disk anyway, to make any difference. And yeah, this is the same thing. It's just this, this directory is causing an alert. So Mooding, you can configure thresholds on all of the metrics and the default way of configuring and alerting of that would be through email. So it's a little bit, a little bit old fashioned. That's the way I've done it in the past. It would be possible, I suppose, to, to send those email to a, a kind of custom SMTP service gateway where you could convert them into Webhooks or something else. But yeah, the standard alerting mechanism of Mooding is using, is using email. Alerting is a general topic. And I think we need to talk some more about, because in the past we've, we've tended to advise folks to set up email alerting for all kinds of things. Email is a little bit tricky for folks to set up. I find in terms of organizing, they're making sure their DNS settings are correct and the DMARC settings and making sure stuff doesn't end up in spam. I'm quite interested in using, using things like Telegram for alerting, because Telegram bots are actually quite a low barrier of entry, quite easy to set up. But it would be a matter then of linking in some of these monitoring systems to your alerting system. So yeah, my answer is I'm not sure. I think I read somewhere that you can do an alerting upload. I've never done it. I think you can. Moonin, yeah, certainly alerting is, is built into it. But it's, it's email alerting. Okay. You highlight an important topic we need to talk some more about is setting up alerting systems of various sorts. Yeah. Cause this requires some manual looking and it's not really an auto feature. Yeah. We look at day by day. Yeah. It's not adequate certainly if you're running a lot of systems. Yeah. And here in OCB, we have a lot of Docker, Dockerized service arranged. So we have at least four or five environments for the HIV. So yeah, it'll be good for us as well. Yeah. We'll probably bring the, the Glowroot stuff is as a, it's just, it's just essentially using JMX and they are all kinds of, of agents you can use to get alerts off, off JMX metrics. I think even Prometheus, for example, Prometheus who will gather your JMX metrics, metrics and you could configure alerting off that. The Glowroot is more, it's more of a diagnostic tool, I guess, than a monitoring tool. I go to, go to Glowroot when I have problems. I usually look at my moon in to see whether, whether my system is, is being well behaved or not. Okay. Good to know. Yeah. Cause you can, you can configure Glowroot within your, your Docker container with your, your Tomcat as well. This means you probably have to make some adjustments to your Tomcat configuration. Okay. So I see the time is up. Yeah. Can I ask a last question? Yeah. I want to know if there is a best practice to set up a password on the morning monitor. There are what? Sorry. To set up a password on the morning monitors. Yeah. You do that. You do that through your, through your proxy. It's not there by default.  I'll make a note of that after just stick to, make sure there's some documentation or how that should be done. We do have a script that does that automatically. We need to maybe, maybe we should integrate that into Tito's Ansible scripts. So that there has a password by default. Yeah. If you just install moon in the basic way, it's going to be open, which is generally not what you want. I can show you how to do that. Maybe, because I'll show you the little, the little configuration snippet that you need to do. Okay. Bob, may I just ask one question? So, who is this? Yeah. No, this is Gerald. Yes. So the question that I want to ask is like, in a case where India is a spike and probably you are not, because like, for instance, it took me a couple of time for me to notice that there was an error that somebody was actually doing a query of the whole database. Right. I looked through the, the, the, the cat from cat log and I couldn't see, I couldn't actually see what, what was going on. But then when I looked through the globe, which I then found out that there was somebody who was doing a huge query and that query was actually taking the system down. So in such a situation, is there any way we can configure notification system? Let's say for instance, when, when the, when the request takes some amount of time, let's say like 10 minutes, it sends a notification that there is a query or something or there isn't, there is something you need to look at. And I think there are a couple of systems now that are integrated with telegram. Is that possible for, if it's not only email, but is there a possibility for us to, to integrate it with telegram so that it just sends you a notification that something is going on with yourself. I don't know if that is possible using a globe or money. Yeah. I think there are two things there, general. I mean, first of all, the telegram thing, I think we, we want to, we want to make it more kind of generic mechanism to make it, to make it easy for people to set up telegram bots for getting all sorts of alerts related to DHSG. So yeah, that is a active area we're looking at currently. Particularly on long running queries. Yeah. That would, that would certainly be something that you could, you could get alerts off. And then, and then yeah, I'm not sure I'm getting a telegram alert to say you have a long running web request. Sometimes a little bit hard because you know, you know that it's run for long after it's finished. You'd have to, sometimes what's easiest to look at in fact, not so much the web request and usually the kind of scenario that you're talking about results in the long database query. One of the useful graphs on the press plugin on moon in is the long running queries. Where are we? You can see here. You have queries here, which take quite a long time. They kind of finish off what I've seen in some cases. And I think you would have seen it in this case here. It's like a triangle like this. And this is a query, which just keeps going and going and going and going and going and going. One of the things we could do on moon in is in a, in an alert here, right? A threshold. If you're query. These are not bad enough. Okay. Some of them have taken like five minutes. You could put in an alert that you've got queries running for more than three minutes or something. You want to make sure you don't end up with too much of that's the important thing, I guess, in your postgres configuration is to make sure so that you log those queries. So that if you get queries running with a certain amount of time that you log them. But yeah, where you'd get the best place of getting the data from. If you want it, if you want it kind of live, you would have to query the JMX. Or you could get it from PG stats table on postgres to postgres will tell you, you know, in a live sense, which query is currently running and how long it has been running for. Or you can get it after the fact. If you look in your proxy logs, the advantage looking in your proxy logs. And again, you could have a script that does this. That would send you an alert if any query has, has run for more than four minutes. Let's say you said then you also get information about potentially the user name, certainly the IP address, things like that. But I think the important thing in your case, Gerald was that this thing had been happening and you didn't know it was happening. If I remember correctly. And that was partly because you didn't have any kind of general monitoring solution there at all. We'll continue to look into it and see what we can do. But basically, yeah, the globe was extremely helpful because we had the challenge. And it seems as if somebody had a third party layer, where in the query were coming from that third party layer onto our system, through the API. And so regularly on a specific period of the day, that query is being executed. And so we were struggling to find out what was the problem. And firstly, the first thing we noticed that was that the, the, the, the alarm that was allocated was not sufficient. And then secondly, we were able to identify the user and then eliminate the user. So that was, that was extremely helpful. Yeah. I mean, often this kind of detective work is, is going to multi-layered. I mean, often in my case, I find Moonean gives me a first indication that something doesn't look right. Glowroot gives me a more detailed, detailed picture of what exactly is not right. And then you look in the logs from there to find out, well, if Glowroot is telling me that they've got this particular request is causing problems. Then I need to go and look in the proxy logs. And then we're going to, Glowroot is telling me that they've got this particular request is causing problems. Then I need to go and look in there in the proxy log to find out who was initiated these requests and whether they're happening regularly, whether it's a machine, what the user agent is that kind of thing. We spend some time maybe in a subsequent session talking about log files and how to, how to make sense out of them, how to analyze them. Okay. We are a good bit over time now. I'm going to have to, I'm going to have to leave it there. I can see a spike just beginning to happen in Ghana. As we talk. Yes. So thank you Bob. That's been a very informative session. So on a lighter note also is that we are planning for the integration academies and serve admin administration academies in Rwanda. In the next, I think two months, March 22nd to 23rd yet. So I'll share the links on chat for those who are going to be interested. Okay. We are, we're totally interested, but for some of us are like, like I mentioned the last time in the group, for some of us it's going to be a personal. It's going to be a personal. It's going to be a personal improvement. And when you're considering the cost, that's almost like $5,000. Right. Because based on all the expenditure that you've been budgeted. Though I'm looking for partner who might support, but is it possible for us to have a discount for instance, for those who are doing both? I don't know. If you're going to do both, it's impossible for us to have discount. I'm just pleading for myself. I don't know. You save on one set of flight tickets. You don't have to get, you don't have to go to two academies. I don't, I don't know general. I don't do the money. Unfortunately. These are the kind of questions we need to raise with Alice. I don't think we've talked really, but nobody's raised this before. The thing I'm having a discount for the two. I'm not sure. I think the, the, the, the, the, the, the, the conference packages. So I'm not sure how much room for manoeuvre there would be in that. Okay. Okay. Okay. If there is a discount, I'll let you know. Nope. No problem. No problem. We'll be interesting if it was a quasi conference like, um, in person and virtual as well. Like have different costs. Structures. Yeah, I think, I mean, what we might try to do, um, because we know a lot of people won't be able to get there. Um, we'd try to do some sessions virtually as well. Um, and I don't think there's any reason to have any cost associated with those at all. So we'll try to have some free virtual sessions. But I don't think we'd be, I don't think we can do the whole thing virtually because it's, I don't know. You kind of lose some of the benefit of being together when you, you're also trying to manage the zoom and everything else. Yep. But we can talk, we can talk more about that in the weeks to come. Thanks Bob for that. That was really interesting from my perspective. Okay. Thanks guys for, for attending. Good ideas or things that you want to look at for next week. Let us know. Um, we're trying to, we're trying to build out a calendar of things to talk about over the months ahead. Um, always happy to get suggestions. I'm interested in what's going on with this. Okay. Thanks Tito. Thank you. Thanks for inviting me. Thank you. Thank you. Thank you. Thanks. Thanks. I don't know. Thanks for everybody. Thanks so much.


Herman Mashaba adds his support for Lvovo's idea for a public meeting on human trafficking. Former Johannesburg Mayor Herman Mashaba has thrown his weight behind Lvovo Deranghos' proposal for a meeting to discuss ways to eradicate human trafficking in SA. The Solilinini hitmaker has been a vocal part of the huge debate on abductions and trafficking that has dominated social media for the past few weeks. After making several comments on the issue, Lvovo said the time for talk was over and South Africans needed to take action. Time for action has arrived. Tweets are great but we need to start acting. All our problems can be fixed if we just unite and tackle them together. We are all we got. No politicians will save you. No pastor or anyone besides us. We will win, he wrote. He proposed a public meeting to discuss the problem and ask his followers if they would attend, arguing that if they waited for the government to intervene many more South Africans would be lost to the scourge. If I propose a meeting to share ideas on how we can eradicate human trafficking as the public would you show up? Fumani Kugavmanye because y'all gonna keep begging until y'all lose all your brothers and sisters. Ever depend on anybody, team up, unite and plan. Enough talking. Mashable was a big fan of the idea and said Lvovo had his support. We are there, he told Lvovo, adding that he knew all too well the struggles of fighting against human trafficking. He said when he started as mayor of Yilburg he was told there some buildings in the city that were no-go areas. Buildings where modern day slavery, human trafficking, prostitution was going on. He alleged that these buildings were protected by politicians and still are. Their comments come as celebs including Laredo Kaganyeko, author Jackie Famatsev, Nakatholomaviso, Samklo and Lvovo, and DJ Franky shared their outrage regarding human trafficking and called on the government to put a stop to it.