Welcome everybody. This is Sartra with Alice Untap. Thank you guys so much for coming out to this webinar today. We've got a great webinar on creating a technology disaster plan. We will also have a handout here available in the next few minutes that's downloadable. As with all of our webinars, we are recording this webinar so that we can put it up on our YouTube channel. All of our trainings for the past four years are up there so we've got over a hundred videos and we've got Joshua Pesca here who has worked with us on security stuff in the past and I'm very excited to see this. This is an update of some stuff that we did about three years ago and this has a very practical focus to it. Thank you all so much for coming out. If you've got any questions there's two ways to really do questions. One is to type them into the question box and I will be monitoring those and I can read them out. There's also a raise hand function and if you use that we will watch the attendees list and then we can unmute you and you can ask questions aloud. If you've got a question there's probably several other people with that same question. Please type it into the box or raise hand so that everybody else can hear that and we can answer those questions as we go forward. Thank you so much for coming out Joshua. Good luck with the training. Thank you so much and thanks again for having me and thanks to our dealer for putting this training together and to reiterate what starts just communicated. Please be generous with your questions. Go ahead and type them into the chat. Go ahead and raise your hand and get unmuted and ask questions. We definitely should have time to cover questions over the course of this and I often find in workshops and training a lot of the best content often comes from the audience in the form of questions and conversations that happen around those. So please don't be shy if you have something you're thinking about. Today we're talking about creating a technology disaster plan and we'll go a little bit outside the narrow confines of technology and kind of general incident response and disaster planning. A little bit about me. I'm the Vice President of Technology Strategy at Roundtable Technology. I've also been an ideal war expert trainer for several years. I'm actually now also a faculty for N10's technology certificate program and basically I've been spending the last 20 plus years in New York City working with literally over a thousand different nonprofit organizations helping in a variety of technology frameworks and Roundtable serves currently about 300 different organizations as an outsourced IT, technology strategy, cybersecurity, website developer, all that great stuff. If you don't know about ideal wear, ideal wear of course helps nonprofits make smart decisions about technology. They do that through reports which are very well researched, different articles, reviews of software guides, very comprehensive guides. They just release one, an annual one that they do on constituent relationship management software and of course they also offer trainings like the one you are attending right now. What we're going to cover today. Scenario planning. What could happen thinking about all those bad things that happen and kind of learning to be unafraid to think about those things. I think people don't want to think about bad things and then they're not prepared for them so I'm going to encourage you to go ahead and think about those bad things without worrying about them just thinking hey what will we do if that happens. We're going to talk about basic four scenarios thinking about you know the initial response like someone kind of saying hey we're having an emergency. Let's let's invoke our disaster plan and officially say that's starting. We're going to talk about reviewing the system, talk about the priority and order of getting things back online and functioning and other planning configurations. That's the basic outline for today and again we've got 90 minutes allotted I think we'll comfortably be with inside that so I really do encourage questions for everybody and by the way in case it was curious I'm speaking to you from New York City I'm six floors up in Queens and I am on a fairly busy street so if you hear sirens or things going by I apologize there's not much I can do about it beyond what I've already done in terms of closing the windows and putting on the noise-canceling headphones but if you're siren noise in the background don't be afraid I'm not having my own disaster hopefully I am doing things and so we're going to go ahead and launch our first poll I think starts I will I will need you for these polls and we're going to ask have you ever worked at an organization that suffered a significant data loss or destruction to the office? We've got about 50-50 so far. I'm not seeing those starts so I apologize I guess I'm only going to see the questions that you share with me as the presenter sorry I'm used to being no problem and I will upgrade you to an organizer just to see if that gives you the additional access. That'll probably do that. Bear with us for a moment if it disrupts the screen shots that's fine and okay yeah and right after I took over LSN Top we lost almost everything with our previous hosting company they had no backups and we had to go through the process of switching hosting companies going with an entire different system and we have since had to use those backups twice in the last five years to recover things from the site so it is very very practical to us very very useful. Yeah and these you know fall into the categories of things that no one thinks about too much until the moment you need it and then I think for anyone who's in the IT field or really need significant data I think all this can relate and I know I've had this multiple times in my career where suddenly an executive director CFO calls you and says I just realized we lost you know accidentally deleted our entire budget for Q3 I need you to restore yesterday's version ASAP and in my own head I'm immediately thinking oh boy I've been getting these backup notifications for the past six months but I haven't actually you know done a test restore I'm not sure I remember exactly how to do that gosh I hope it works I hope these backups are in good shape and you know the heart palpitation that everything are happening and that's not a feeling that you have to have in that scenario if you you do a reasonable amount of planning and that's become your whole life do it right so there are so many different kinds of disasters and the first point that I want to make to everybody is attending today is to not try to anticipate every possible thing that could happen a that's not possible in any practical sense and be most of the different bad things that can happen still have a limited number of ways in which they impact you and we're actually going to give you some kind of different scenarios to help you kind of reveal the different problems that different kinds of disasters may may produce and so I encourage you not to you know get overly concerned about so and we're gonna go through some of the kinds of natural disasters here the first of all the flood and what happens if water comes in if you're a ground floor basement if one thing that don't think about the comfort of another slide but fires often more of the damage is done from water from the fire department in fires than is actually done by the fire and floods can happen because of a fire so even if you're many floors up you still may want to think about what happens if the sprinklers for the building got engaged what happens if you know people put a bunch of water in here the floods is one thing to think about earthquakes and these are things where you may have power outages you may have fires you may have disrupted access to your office you may have people stuck in the office and unable to leave and there's lots of different kinds of scenarios obviously here in New York following 9-11 and then again following Hurricane Sandy New York I think overall does a pretty excellent job as a city and organizations within the city overall do a pretty good job because we've had some pretty significant things that have really tested our disaster planning and caused organizations to really think hard about what to do in these scenarios but think about that kind of thing and then you can think about tornadoes what if things get picked up and you can kind of keep going through this and make yourself a little crazy it's like fires tornadoes you know earthquakes you know what about tsunamis what about you know and then of course then there's hacking and then we could also think about in terms of cyber security just flat out deletion of data failure of hard drives employees stealing stuff you know theft of systems within the organization all these different kinds of things that can result in disasters or significant incidents that you need to have some kind of response for and ransomware is another one this has happened to lots of organizations and this is where backups can can obviously you know play a huge factor but also thinking about time and how long your downtime will be and things like that and rather than well let's go to school first so let's have people type in having gone through these and I'm deliberately not spending a lot of time here because I'm going to spend more time on what's coming up but I'm curious if people can enter into the chat again for me what kinds of disasters you're most worried about for your organization and I'm going to delete the stuff that's here already just to clear the path for for what people are most worried about so go ahead and put in what you're most worried about we've got a hacking ransomware hacking hacking time unless for folks very much in the news so hacking and staff errors is one thank you everybody for putting these in by the way hurricanes and hacking so we probably got someone who's living in hurricane stuff got someone who's concerned about earthquakes and hacking and staff errors so a lot of a lot of focus definitely on on hacking less so much on natural disasters seems to be the real concern so I will certainly not give short shrift to the natural disaster concerns but clearly there's much more consistent concerns around hacking and cybersecurity concerns than the other thing so I'll make sure that we give ample time for that and thank you all for for doing that so rather than think about each of these different kinds of things there's some broader scenarios that I encourage people to think about in terms of disaster planning and these won't cover the cybersecurity piece but they will actually I'm sorry one of these will but they'll give you if you do these as kind of thought experiments ideally with executive staff within your organization who can give clear responses as to what's truly important and what's truly not within the organization these scenarios will help you cover most of the kinds of things that happen and Brian I'm I don't imagine that you've seen these or maybe you've reviewed the deck but I'm curious to your feedback on these as well and I've used these when I do strategic tech planning with organizations and we get to our incident response these are the scenarios that I I'll literally have folks walk through and I'll talk to the executive directors and to say I just want to spend 10 minutes on each of these and I want you to really think about this and think about what would happen so the first one is it's not intuitive at all but just take all the all the technology out of your office all the computers all the servers all the network equipment everything and replace it with brand-new stuff so you leave the office one night you come back in the next morning and you know I we could say this is good news all of your old seven-year-old computers are replaced with brand-new ones your old servers replace the new ones you have a brand-new firewall brand-new switches brand-new everything and the question then is what what does our recovery path look like to actually being operational from that point we've got internet we've got all brand-new equipment everything fabulous but we're going to need someone to get it all working for us you know get a connect to the internet and network and then we're going to need someone to recover the data and sort of get it operational again and this little thought exercise helps you understand what your recovery time is really going to look like and also what information would be lost in that scenario if you say to yourself oh my god you know all of our backup tapes were actually in the office so if you're telling me that someone came in and replaced all of our existing backup tapes with old backup tapes that we'd have lost all of our organizational information for the history of time well you've just revealed a pretty in my view significant you know risk of your existing environment right that that you don't have any kind of off-site recovery capability and this scenario I find is a really useful one as a kind of starting point to think about and it also helps you understand another thing I find really often and especially with executives not to pick on executive the system is understanding if the executives for those of you here who are IT folks is really important executives often equate a backup the phrase backup with high availability and for those of us that are IT folks we know there's a very different thing that I could have if I have let's say a file server with a terabyte of data that is serving files with permission sets and all this good stuff to 50 different people and that file server stuff is a catastrophic hardware failure and I don't have any other server to which I can restore this terabyte of data and my backup is in the cloud and may take you know three days to download a terabyte of data and then I'm going to have to figure out a way to acquire another piece of hardware on which I can mount a terabyte of data and share it out with decent performance and perhaps even you know recreate all the shares and remaps them it might be three or four days of downtime and what I find is executives don't understand that they're like well we have a backup why can't it just turn on and it's really important to communicate that and the difference between those two things to executives so they understand the next area because it emphasizes the need to have that backup in a second location and then also practically run through once every year or two years going through that process and seeing does it work how does it work what is the downtime is that acceptable do we need to redesign this to change that and if you do that proactively before there's a problem when a problem comes up it's so much easier to recover yep and and I keep people it's really easy to make assumptions that won't be in alignment with your executives and I've made this mistake at my peril and learned from it where I've assumed that they don't want to pay a thousand dollars a month for a high availability solution when they're paying fifty dollars a month for crash plan right now and been dead wrong about that and when I explained to director like in our current scenario this system this you know your financial and donation management system is going to be down for three or four business days if we suffer a hardware failure they've said what's the cost to make that not true I've said about the thousand dollars a month I said let's do that that's a no-brainer for me so don't make assumptions about that you know the cost is going to be prohibitive all right second one is the power just take the power out for a couple weeks and figure out what's gonna work at your organization and what's not one of the scenarios we'll look at later actually that happens and and they made some changes based on that and I following Hurricane Sandy you know my running joke I don't know if this joke is right but I I've been a big advocate for cloud services ever since Gmail and Google Docs came online and then Office 365 and then they started donating it to nonprofits you know going back even 10 years I've been a real big advocate encouraging organizations to move away from in-house infrastructure to cloud services and really getting not a lot of traction with that as nonprofits were really comfortable where they were and were somewhat suspicious of the cloud and Hurricane Sandy really was and I don't mean to be glib about Hurricane Sandy but it was in many ways the best marketing for cloud services here in New York City that could have been produced because the amount of migration to cloud after that was profound as people realized oh yeah that's why cloud services are really helpful because when the people that couldn't go into their offices for two weeks but were using Salesforce and Google apps now G Suite they were like okay well just work from home or work from this other office that someone's running us for the week and we're all fine and that that's a real powerful thing so take the power out for a couple of weeks think about well what systems are we gonna have to go in and pick up a server and carry it to some other location and power it on if so do we have a plan for that do we know how we're gonna get there what if we can't physically get to the office we have any way to get that data if we're denied access to the building because it's in a disaster like that so that's another scenario I find very helpful to shut the power up next one the the class pointed tip by the bus I think people have decided that means and now the one is they win the lottery and they move to Aruba and they never return our calls again but just go ahead and take some senior staff out of the picture they leave unexpectedly you can't talk to them anymore you can't get any information from them what what what have we lost and how much disruption is there in terms of institutional knowledge about different things there's a lot of organizations that are at massive risk of this with internal IT staff where there's not good documentation they're the only person that knows the passwords for things no one else to be able to get into it and it would be really problematic if that person was unavailable to the organization this is something just just experientially over the last 20 years happens less than the first two the first two scenarios are more common at least in some portion than this scenario but this scenario does happen and it's certainly worth spending a little bit of time to think about what redundancies you do or don't have Brian do you have anything on these last two before I start my first few months of taking over LS untapped it was a matter of interviewing past staff and finding passwords and creating lists of resources and who controlled the domain name and all that type of stuff if those individuals had died and they were not around a whole swath of access would have been lost entirely just creating those best practices to keep those lists accessible available to other members of the staff that type of stuff can save you hours or even months of work if there is an issue and to add to that I would say the a really common thing that I do see and I mean at least several times a year is some business critical service and the most common one that people can think of as your domain registration so LSM tap org is registered to Brian and Brian leaves LSM tap and has gone for three years and the domain starts to expire and all of the notification messages from network solutions are going to Brian at LSM tap org Brian of course isn't responding that email isn't there it's getting on replies and the next thing you know the website's down the email's down everybody's totally confused and it turns out that the domain has expired because no one was getting those notifications and that can happen for other business critical things like SSL certificates website hosting contracts phone vendor contracts all kinds of things and where you register these things try wherever possible to use distribution lists that are role based as opposed to actual individuals and if you are inheriting an environment going through and getting all those set up is a really nice proactive thing to do. Yeah we had a program both lose a YouTube channel because it was registered to an individual's personal Gmail and a domain name all within the past six months I mean these are very practical pieces of advice here. Yeah and that and that I just want to tell everybody that is such a common thing that I see and and it's just you know it's just it's a little bit of work to go do it takes maybe you know 15 20 minutes for you okay what's that account but it saves you so much headache in the future and boy I'll tell you there's nothing worse than having and I by the way I've seen it worse sometimes that the domain expires and then someone else snatches it up but now you're buying it back from someone who's very opportunistic that happens less than it used to but so there all right so scenario four so we got scenario one is we have all new technology yay what data did we lose right because all our technology is just taken from the office every computer backup safe laptop you know phone everything taken replace brand new stuff what does that look like second scenario turn off the power for two weeks third scenario get rid of some key staff they just leave and they don't talk to us anymore and then the fourth scenario now we're getting the cyber security and this is just imagine that some sensitive information is breached so let's say you're you know Salesforce database with all the donor information so security numbers and contact information is breached bunch of questions here how are you going to know what that's happened do you have any kind of system of breach notification that would possibly help you with that and then do you have any kind of communications plan do you have a template written up of something that you would communicate that would do this really clearly this this is a wonderful thing there's wonderful examples out there of a appropriate breach notification that is communicated because this is something that pretty much every major company has either already had to deal with multiple times or it's going to have to deal with and there's a bunch you know a checklist of things to communicate to your customers or your constituents or your staff to say you know here's what we know here's what we don't know here's you know what's happened here's what we've done so far to try to remediate it here's what we're going to continue to do here's what if any action you should take as our donor or as our constituent and you know if you have questions here's the thing and there's if you do this well it doesn't it doesn't have to result in reputational damage for your organization and in some cases it can even enhance it you know I I've been approached by organizations where they had a breach and I said look the way they handled this everything and it only speaks more highly of them like the fact that they got breached does not speak poorly of them by itself because anybody can get breached and Brian and I've done security things and you know that's that's clear but how you respond to that that's very different so there's a lot of prep you can do there to just have yourself in a better response so those four scenarios I think are really nice things to just run through and and we'll cover a lot of the kinds of other ranges of things from a technology perspective again we're not that that's not going to cover all the human concerns around different disasters but that's that's a bit outside of technology but I think that those four kind of thought exercises really cover an enormous amount of ground in the disaster planning area right and having policies in place that control what information you actually store and then when you destroy information we've done a bunch of stuff recently on data destruction policies for n-tap help limit both your liability and the possible harm when it comes to those type of breaches they will eventually happen and not having the information the sensitive information unless it's absolutely needed is going to help significantly in those cases yeah and Brian I don't want to get too far a field but I actually just completed a project here in New York City for a civil rights organization that had very significant government subpoena concerns and they actually asked me to decide something where I would say I wouldn't give up information if subpoenaed by the government then I would do which I had to say I'm not you know I can't can't sign that I'm not going to go to jail on behalf of the organization but I'd be happy to work with you but we so we have to set up a very clear compartmentalization structure where sensitive information that covered certain areas has never touched the roundtables environment ever so certain things to review I just did it within their office looked at it only within their office and on their machines and never put it on anything so I can legitimately say if the subpoena comes to me I don't have any of the information that you're asking for like yeah I looked at it but I don't remember what any of it says you know blah blah blah and and not having information that is sensitive that you don't need to have for business purposes is in many ways kind of rule number one of cyber security right the easiest way to protect the information is to not have it in the first place if you're worried about it being breached so that's an excellent point right same thing is true with even internal employees create a system to where you are only giving them access to things that they essentially need that way if their information their password is is compromised their only your only loss of information is limited significantly yeah and as a in the cyber security area I think it's like intelligence subscript a principle of least privilege or PLP which is just you know that you operate from the realm of people only have access to things when they have a business purpose that access to it and then they have the least amount of access that they only read only they only get read only you know so on and so forth as we think about our first response all right first things first we want to make sure that our personnel and our constituents and everybody is safe and I think it's really important to understand that while there's obviously technology needs that go on people's personal safety comes first and I do want to be clear having been in a number of disasters this is something that is easy to lose sight of when you know the proverbial stuff is hitting the fan and I encourage people to make it explicit in your disaster planning that that the understanding is that we're not asking staff to risk their life you know to protect organizational information that that personal safety of employees is paramount unless you know you're in an organization where where mission-wise that's that's inconsistent with your mission where where you're doing work where it's understood that you're potentially putting your life in danger in order to serve the people that you need to serve but I would say for the vast majority of us especially in legal services you know you shouldn't be risking your life or putting yourself in personal danger during a disaster especially to perform some kind of a technology or recovery role also just having an idea of you know how are we going to how are we going to verify the safety and contact of everybody how are we going to know this was uh after 9-11 here in New York this was a real challenge and really you know petrified a lot of different organizations I worked at that time for an organization that was not you know within blocks of of World Trade Center but was about a half mile away and you know there were staff that were not there that day it was very difficult to communicate a lot of the digital infrastructure was down and and it was hard to verify the safety of people as we were going through and that and we spent wind up spending a lot of effort you know probably two days before we were actually able to verify that all about 45 of our staff were okay and you know we're in a place where they were you know able to be safe and then we started working through well who's online and can communicate who has a phone who has email these are some kind of basic things that it's worth sort of figuring out as quickly so you can kind of have a dashboard to know who's who's available who's not another thing it's not this is declaring that something's going on so if you've put together this plan and says okay we've got this team here's what we're going to do here's the steps we're going to take someone needs to declare we're doing this this is now everybody who's on this team who's on our incident response team this is now your your job until we declare we're back to a normal state or we're resuming some level of normal functioning and again i really encourage organizations to make this explicit and and have a little conversation ahead of time where everybody understands that like if there's a five person team that involves our IT director and our HR person and a communications person and our executive director and say okay these people are on our incident response team then when if we declare an incident if the executive director says okay this is an emergency then that's everybody's job until the executive director says we are now resuming some form of normal operations and now your jobs are shifting again and that's important understanding how you're going to communicate if the office is down especially if you have like internal email systems internal phone systems if if your communications infrastructure is going to be down and in one of these scenarios like your power then understand how you're going to communicate with people after that and this is where people make these mistakes that they have personnel files that are all digital in the server that then can't be accessed so we're down we don't have email we don't have phones the emergency numbers for everybody is on the file server that's also down so now we can't get people's home numbers or emails and now we're just scrambling and trying to get all this on the fly which is obviously not an optimal way to do that having emergency contact trees and directories that are stored outside of your normal system so that they can be accessed in the event of an emergency really important with the cloud world this gets a lot easier but i've seen that mistake made a fair amount defining roles having roles having backups for those roles who's responsible for communicating with the staff who's responsible for recovering the systems who's responsible for you know communicating with external parties with donors who's responsible for updating the website to let us know there's a lot of different things that need to happen and even for for minor stuff just to give everybody a little perspective we changed a workflow at roundtable about a year ago where when we have emerged so we do emerge you know support tickets we run a help desk and we provide support to about 300 different organizations and when we have emergency tickets come in like a network down and you know there's some kind of hairy work that has to be done in the firewall or someone's you know suffering a DOS attack and we're having to figure out how to deflect that you know our engineers are really you know heads down feverishly working on the problem and in the meanwhile there's some executive director who's asking for status updates and we learned that it was completely unreasonable to ask the senior engineer to be putting together coherent communications to executive directors or CFOs of organizations about the status of this emergency and we started assigning project managers to emergency tickets or you know to system-bound tickets whose role is simply communication so they just talk to him with the engineer and bother them for 30 seconds and say I just need to know a couple of basic things just answer that for me and then I'll take it from there and then the PM is responsible for a communicator that's really important because you don't want someone who's dealing with an emergency to have to take time out to think about communications you want someone who's dedicated to just the communications role and thinking about all those different roles. Brian do you have anything you want to shed chime in on here cruise along. No I think that the defining role roles is important and once again actually practicing these things especially when it comes to the previous point about checking in on staff making sure everyone is safe like set up once a year or once every two years actually going through a scenario writing it up and practicing it with staff so that you can figure out whether what you put together in theory works. And I think the great balance also come up multiple times is you know how much time we spend in disaster planning versus we all work to do right disaster planning it's sort of risk mitigation it's very unsexy it doesn't improve anybody's life unless this really bad thing happens and then it has a potentially huge impact and these things are always very hard to determine how much effort to put in and I encourage you not to go crazy with the amount of effort it's not something you're spending weeks on but as Brian I think keeps mentioning I think as a once a year something you maybe take a day and just run through the scenarios with folks look at the plan that you have if you don't have a plan you'll have to spend a little bit more time than that to develop an initial plan but once you've got that it's really just a matter of updating it once a year and that's sufficient but do it do update it once a year key typical roles executive director is the one who's going to declare the emergency who's going to make you know set priorities they're obviously you know the person who's calling the shots you've got the IT director who's going to you know take care of the technology things you've got operations you're dealing with the building and dealing with facilities you've got HR dealing with the staff you've got a program in terms of services and then communications who is communicating with everybody those are a pretty good breakdown of the different roles that you may need and depending again on your organization you may not need all of these things or you may need more but it's important to understand the different functions that will need to be happening during emergency and making sure that you have appropriate personnel assigned to them and that they're not overburdened with the things that they'll be needing to do as brian mentioned keep a directory this should you know have all of the emergency contact information for staff the personal contact information their personal mobiles personal emails so that if you have organizational systems that are down you'll still be able to communicate again with cloud services if you're fully cloud-based then you those systems should continue to operate in the event of an emergency and and that that makes life a lot easier this is not by the way the time to realize that you actually don't know who your phone vendor is or who your website provider is or how to update your website or who your service provider is or who your building manager is these are all things you want to have contact information for all of those and have some kind of relationship with them just you know have actually called them and verified that they'll actually answer the phone at that number and things like that and that's a that's a task that can be done as part of this is just someone actually you know calling those numbers emailing saying hey is this still the right contact if we need help with this or you know this other thing keeping those records updated establishing a meeting place this is you know kind of you know disaster planning 101 for families and for businesses if we have to leave our home because of a fire if we have to leave our office because of fire if there's an earthquake and or no one can go to the office then where will people meet do we have an alternate alternative meeting place that can function as an office and what's the level of functionality we have there if if you're just looking for a place to meet it can be in many places a public place that's within some distance of your office and if you actually need an alternative working space then obviously that has much more stringent requirements for what you'll need you'll need desks you'll need internet access from the computers you'll need all that kind of stuff and you know obviously a backup for that so if your meeting place was three blocks away and the disaster is an earthquake that shuts down a whole city then you need some other plan for that and this is where i i don't encourage people to get too nuts about this stuff in terms of you know having like some sort of off-site facility in another state where you're going to go meet but at least kind of say well what are we going to do and the default stance for most organizations is people are going to work from home which is i think okay but again it depends on the functionality of your organization we're going to get to that as we as we proceed here and just as a reminder for people under the handout section there is both a copy of these slides and there is a two page little takeaway that covers all four of those scenarios and then all 20 plus questions that we're asking here in a quick checklist so i recommend people download that checklist and use that share that it's under an open license please customize it make it practical for you also and then kind of basic needs for folks which is you know food shelter movement a big one that happened after 9-11 i'll tell you a big change that happened in New York after 9-11 which is pretty much every female employee at every office in New York City keeps parable tennis shoes under their desks because people who had to walk 100 blocks uptown quickly to escape dust in high heels were really you know kind of like that that was a disaster and and that's just a really small easy thing to do that can actually turn out to make a big difference if you have to run or if you have to walk a long way or may not be able to get to another pair of shoes in some period of time that's a super cheap easy thing and guys can do that too obviously but guys dress shoes tend to be you know reasonable for for long walks or or mild runs other things like water food flashlights there's a lot of pretty basic stuff that's easy to have on hand and inexpensive to have on hand that can turn out to be really handy if if you have some kind of power outage so just dealing with that kind of basic stuff again this is not technology stuff at all this is just basic so just having little you know a bag that people have in their desks with bottle of water and a granola bar and a cheap led flashlight and a pair of shoes you know that that's certainly not a bad idea to have all right we'll ask people to put this into the chat again how confident are you that you'll be able to reach everyone associated with your nonprofit in the event that you they have an emergency are you very confident are you not confident are you very unconfident so we'll ask people to put in there so you can put a yeah go ahead and put a c for confident you could put a v for very confident and for not confident let's see people or people who's putting the letters for there yeah that's good but you guys are so smart you're putting the letters for the thing yeah so c not very confident not so people are not kind of a single a here no one seems very confident it's everybody's somewhere in the middle there and that's an indication that there might be some work to do to make sure that you you know can get in touch with everybody in the event of an emergency and again think about the different parties the different vendors the different facilities people this you know in the middle of a disaster it's not the time to start forming a relationship with a critical vendor that you have it's much better to have that relationship ahead of time all right thank you everybody all right so we're going to go into review your systems and this is where i'm going to start thinking about the the level of you know the acceptable levels of risks for organizations and how this can vary a lot and this is really where in my view a lot of the time that you're spending around disaster planning should be focused on is in conversations with different stakeholders within your organization to try to understand what are the priorities of this organization in different scenarios and what's acceptable levels of risk for us the backups versus high availability the $50 a month backups versus the $1,000 a month high availability solution and the acceptable level of risk against those costs are important kinds of conversations to have before you're in the scenario and your executive is deeply unhappy with the risk that they didn't realize they had assumed and and that's you know for all of you who aren't executive directors you're on this or aren't in an executive role that's like i can't emphasize that strongly enough to really try to have those conversations ahead of time we take an example and not to in any way disparage the importance or diminish the importance of community theater to a civil society but if there is a disaster and you're a community theater it's probably not critical that your operations continue you know that day or that week it's probably going to be okay if if you guys aren't able to send out your you know event notice for the month of july this friday if it turns out there's some major natural disaster in your area and your your theater is off so that that's one on the other hand if you are a domestic violence shelter and you're taking in victims of domestic violence and their children or children who are you know victims of then your services may be even more critical you know and certainly no less critical and your ability to perform the various functions that your organization are still just as necessary in the event of a disaster in a very different kind of scenario than the community theater and obviously there thousands and thousands of different kinds of organizations so thinking about for your organization when stuff starts happening what what do we need to be able to do who's depending on us how much are they depending on us and what's our ability to do that with different you know in these different kinds of scenarios that we that we thought about we'll throw another poll here in here how important is it that your organization continues providing some of your critical services during a disaster so if you give a one here you're saying hey we could close for a few days our whole business could close and everything would be fine so if I take roundtables an example right we're probably I'm not going to put us at five but I'm going to go ahead and say we're probably at a three and a half or a four in the sense that again 300 different organizations count on us for their IT support and and need us in order for themselves to be functional and some of those organizations are in fact domestic violence shelters and other organizations that that provide much more critical services than we do but they're dependent on us for their information system so for us we you know if there's an actual disaster in New York or in Maine where we have operations we we pretty much need to keep functioning at least on some level we couldn't close for a few days and everything would be fine but but some of you are probably are in that scenario and most of you are kind of in the lower I don't see anybody putting higher than a three here so that's good that makes a more relaxed kind of situation for you in terms of your your disaster recovery think about your essential functions you rank the activities of your organization a lot of time something like payroll is really high I'm making sure that payroll gets processed is very high function making sure that you know all the vendors get paid and things like that basic functions a lot of times people think of email to critical service a lot of times it maybe is maybe isn't again you know if you're domestic violence if you're providing legal services to people in crisis then some of the services may be the most essential something or less again when I have these conversations with executive directors it's I've never not been pretty surprised by what they said were the critical services and again my assumptions from an IT perspective have been pretty consistently wrong about what what I saw for the critical services and then what the executives in the organization said all I really care about is like this folder and this application like all I really care about is our financial application and our HR folder I don't care about anything else in our system I don't care about email I don't care about the whole rest of our document management like as long as those two things are able to function and you know if you get a really clear response like that it really narrows from an IT perspective what you need to apply high availability to and and where you can kind of you know exert effort and then if everything's down you now have a very clear priority list without having to in the middle of a crisis be talking to people to try to establish a priority list you have one that you're working from already so in the event that everything's down first bring up the financial system then bring up the email then bring up the phone system then bring up that's the order in which you're working on restoring services if it's been everything down having inventories is obviously going to be very helpful knowing what you have knowing where it is having an understanding of what's online a thing that I found very helpful are checklists of you know to figure out the top 10 or 15 things that are working so anytime I have a large complex environment and we have to let they restart it or there's a power outage to the building which happens sometimes or there's you know a power outage to the neighborhood we have a little checklist of things that we run through to test the different systems to make sure that everything is back online the way we expect it so we don't find out three days later oh this one system we rebooted and it really didn't come back online the way we expected this website isn't working or this application isn't working the little checklist can be hugely helpful here to make sure when you power everything back up you know you just run through the checklist matching the equipment to those functions pretty basic stuff just helps you prioritize what to fix what to get back online so obviously you know internet that's an important the power comes first then maybe internet then you power up the phone system and other things so understanding your sequencing systems and what comes online first that that will help I'm sorry I have to I've made it all the way this far without having to minimize my window so think about processes if you don't have you know everything how can you get the job done and are there other ways that you can accomplish different tasks if you don't have the particular tool so again if your if your office is shut down because of power and you're primarily on cloud-based systems then again your your staff can ostensibly work from any location and work out their personal computers and still be highly functional potentially same thing if you're in cloud-based voice systems if you're using something like dial pad or ring central or eight by eight or something like that you know even the phone system can remain fully functional in an offline people can still use the regular numbers you can still receive the regular numbers but think about different ways you can do stuff a big thing that I see organizations struggle with are the outbound greetings and website notifications in the event of crisis so if you have something to take your office offline and a lot of people call your organization then often no one's quite sure how to change the inbound voicemail to say oh we're offline and please call this number people aren't sure how to reroute the number if it can't pick up again this is so much easier if you're on cloud-based voice systems but making sure that you have documented all the processes and know how to do those things ahead of time so you're not trying to figure out on the fly this is that high availability option so if there's something if when you run through these scenarios if you discover something that you really can't be that can't be offline for five minutes or ten minutes or a day or two days then thinking about replicating it and making sure that you have that available somewhere else and then getting into cyber security attacks and we'll take a little bit of time on this because so many people talked about with with cloud systems there's wonderful notification tools that you can set up for google apps or g-suite now office 365 sales force dropbox etc that can help you know if for example uh brian i'll just use an example again let's say brian's on roundtables google app system and suddenly his account logs in from uganda and that's not something that he has done before i'll get a notification that'll say hey this account just logged in from uganda and i can look at that and if i don't know for a fact that brian's in uganda or remoting to some system in uganda that he might be checking his email account from then i can reach out to brian say brian this this happened it's making any sense to you he says no then we can say okay we've had a breach we have a breach of brian's account let's go change his password let's go review what information was there and we we've gotten immediate notification and we can start to think about containing that in terms of malware you know obviously to get ransomware there's there's a pretty set of things to do for an additional resource as part of the ninja series that that brian helped me out with the privacy section of the the ninth class for that you can reach these at ninja.rtq.nyc brian i don't want to start browsing right now but if you want to go grab the url for that the ninth one we did was infinite response and there's a guide put out by digital guardian which is the infinite responders field guide which is was was released in march of this year and is the best thing that i've seen on this topic in terms of being really readable really understandable really clear about the process to go through and and how to contain you know understand that you've had a breach contain the breach meaning let's let's make sure it's not getting worse then let's figure out what you know it's the best we can what happened what we're going to do about it and start to communicate externally and and do all these things and so that for ideal where they review that here i encourage everybody to have a webinar if you want to go watch that but at the very least you can download that digital guardian uh incident responders field guide which is very good around these kinds of things and if you aren't if you're using these cloud systems for breach notification i really encourage you who either can talk to your it admins or who are it admins to really explore all the different alert notifications you can set up in these cloud systems i don't want to get too deep into this topic now and brian you can let me know if we want to explore this or folks you can you can put questions in the chat if you want to explore us deeper but one of the real challenges around cyber security is actually knowing that something's happened that you've been breached and historically this has been something that's been kind of limited to organizations that have a lot of resources because you'd need all these logs you need to review firewall logs you need to review system logs you need you need either an application that looks at those logs and breaks down different reports or you need you know cyber security personnel who actually know what all these things mean and even with those resources you have a tremendous signal-to-noise ratio problem because if i go review a typical firewall log there might be a thousand intrusion detection you know notifications within the firewall that's just a normal day on a normal network that you know there's all these botnets and everything that are just constantly poking and prodding at any public IP address that's out there and learning so when i say a signal-to-noise problem i mean if i've set up alerts for these systems i'll be getting a thousand alerts a day or 10,000 alerts a day and it becomes meaningless it becomes very very difficult to to parse any meaningful information from that but what the cloud providers have have done over the past year and are continuing to do is allowed you to really kind of tweak and set notifications for different kinds of events that are actually meaningful and actually don't bury you in alerts that aren't helpful to you and spending some time to start setting up those alerts and monitoring them can be huge and helping you understand that something bad has potentially happened Brian is there anything you want to throw in the chat? I dropped a link into the chat for the digital guardian incident response web page there definitely worth checking out i'll drop a link to the cyber security ninja series here in a minute into the chat also thanks and the main one that i just in in terms of this is the ninth session which is on incident response and was based purely on that on that guide so looking at that guide will give you a lot of that but if you want broader cyber security stuff there's a lot of stuff there and Brian thank you for your help with that what about paper thinking about that if you have digital copies of stuff you know that's obviously better there's all sorts of ways to convert a lot of paper stuff to digital if you have needs do that if you don't need the paper get rid of it have it shredded if you do need to keep stuff really consider having it converted to digital if it's really important you know you can warehouse it you can do other things but if it's just sitting in your office and filing cabinets and it's actually critical information to have and you have digital copies of it then it's just like any other thing if we go through that scenario right if you have a fire if you have a flood if you have some like that that information is potentially all gone with no ability to recover it and that would be a bummer all right into the chat so for you folks i'm curious just what your quick thoughts are what's if you hadn't outed let's say we just took all of your systems offline for the moment and and it was going to take you let's to do this as a kind of clear a thought exercise let's let's run it like this if i took every every piece of your organization offline and said it's going to take you two hours per thing so to restore email take it two hours to restore file sharing two hours to restore your phones two hours to restore your you know CRM application two hours what's the first thing that you would want to recover for your organization i'm wondering if that's clear for folks or if they're you know kind of taking guess i'm just curious as to what people would put in so i've got one vote for email email would be the first thing that they would come back let's see if other votes come in here um and while people are um responding the url ninja dot rtt dot nyc has links to all of the past sessions um so all 10 of them are there including incident response which is number nine um wonderful sessions they're each uh rather short compact a lot of information in them from a security perspective i strongly recommend running through these we're going to do a blog post for lsn tab highlighting these at some point in the near future also yeah and there's there's no need to watch all of them just to be clear there it was designed as a series of 10 but but it's absolutely you can just go pick oh that's the topic i'm interested and go look at it so thanks Brian all right so we've got mostly email people are mostly concerned about email and that seems to be the number one thing we've got another person saying file maker databases and so that's an interesting thing because that's probably running on a server somewhere and then you need to think about that you know if that server got wiped out in a flood or the hardware failed or you know things like that then you know how would we recover those things another thing and i don't think this is covered in the session is to understand the concepts of different levels of downtime for cloud systems just so everybody understands there's these concepts of five nines and you can deal with the pedia page on it and i'll i'll use the numbers that i have in my head these might be slightly off but they're roughly accurate you have five nines is the kind of gold standard industry for uptime and what what five nines means is that a system that has five nines uptime is 99.99 percent available meaning there won't be unscheduled downtime and that's that means there's i believe less than five minutes per year of unscheduled downtime for a system that has five nines available that's about as good as you're going to get pretty much all of the cloud systems that that most of us would use office 365 g suite dropbox box sales force that are run 39 99.9 percent and what that translates to is roughly eight hours of unscheduled downtime per year it doesn't sound too bad and for most of us is not too bad however it is important to understand that that is unplanned and if there is a particular time of year where not having access to your email even if some gmail would be like a existential level threat to organization like would be catastrophic then you may need to look into either paying you know some of the providers are starting to offer five nines availability as a tier of service that you can buy your buy your way into and then they they have you on a different platform or you may have to go to a provider that can actually guarantee five nines this is a bit minutiae but i do think it's important for folks to understand that while cloud services are likely to be much more available than anything you're going to be running out of your own data center they still are not a hundred percent uptime systems they still do have downtime and in my experience with with most of these it's usually not complete downtime it's usually the service is degraded so google drive will be kind of wonky for lack of a better word for a couple of hours on a wednesday afternoon randomly and outlook web access will be wonky for lack of a better term for a couple of hours or there can be major internet disruptions i don't know how many people are around for the dine dns which happened i want to say about six months or eight months ago where a major dns service got ddost and and basically taken offline and that resulted in huge swaths of the internet kind of not functioning well so a lot of people kind of basically couldn't get to their websites or couldn't get to gmail or other online services and there's lots of other things that can happen amazon s3 service runs way more of the internet than virtually anybody knows and if that hasn't outed and a lot of things down so it is important to understand these cloud services can be disrupted and if you have like no backup plan for that that's that's probably not a great situation especially if you're email as you're saying that's a really business critical service all right so into our case study her justice sandy was on the on the way we talked about this mario shonzi is someone i've worked with regularly quickly made a plan by the executive director is the wrong title for her she's director of it made a plan a couple of days before the storm struck and i remember because i was on the phone with her at that time and we decided that the best thing to do was that they was very likely that they were going to lose power and even though there were many many floors up but they was clear that they're going to shut down and they had a number of internal servers virtual servers all these other things and we decided that it would be best to preemptively shut those systems down this happens in a lot of systems new york city subway system does this as well if they know a big storm is coming their recovery time is you know exponentially better if they preemptively shut down the system and pull the trains out and turn off all the electricity so if there's electricity and trains in a tunnel that floods then all of a sudden they've got all these switches and all this electrical stuff and these stranded trains and it takes them days or weeks or months to get all stuff out but what they've learned over many many years is if they know the flood is coming and they clear out the tunnel and turn off all the power to it ahead of time then all they have to do is wait for the water to recede and then turn everything back on and everything's basically fine and that that was actually how they did things with sandy and they were back almost a full functional in the subway system very quickly so she did that and it flooded and they closed the office for a week so they were I think they weren't allowed in their office for three days they decided to close it for the whole week and when they came back everything powered back on and everything was fine you know which is fortunate they they were one of the more fortunate where their power wasn't out for a longer period of time their internet wasn't out for a longer period of time and because they preempt they didn't have you know unplanned shut down of their systems there was no problems with bringing everything back up everything was fine however they at that time because they had to shut everything down their email and their document management was all on those surveyors and everybody was just communicating over personal emails during that week of outage time and they didn't have access to a lot of their organizational files and they there's something that accepted and they knew that was going to be the case but they said okay if this happens again we'd really rather be much more functional and that was as I referenced before the impetus to migrate to cloud so they migrated to office 365 and sharepoint and one drive for a lot of their stuff and that's you know a really common thing. So getting back online if your hardware was damaged how you replace it how long is it going to take is your insurance going to cover it you know do you have a plan for getting it so just having basic questions like that I would say the insurance thing is a big question. Thinking about where your data is the thought exercises that we ran through in the early part of it the first one of removing all the technology from your office and replacing with brand new stuff and then the second one of just turning off the power for a couple of days that that helps you those two little scenarios really will help you understand where your data is because if you really go through those and with your IT person who knows where all the stuff is you'll know what data gets lost when you remove all the technology and you'll know what systems are unavailable when your stuff is down and so that's important to understand going in really basic backups know how to do recoveries do recoveries regularly review your backup selections most common things I see with backups are obviously people not having tested and turns out the backups aren't good but I would say that the most common backup fail that I see is backups the critical data not having been selected to be backed up in the first place easy scenario to give you organization has a backup system everything's great they use QuickBooks they decide to migrate to Fundy Z they had QuickBooks on a server and the backup was pointing to the QuickBooks folder was backing up all the QuickBooks company files the backups of the company files got ran to another folder the backup scrapped those everything funky door in and they migrate to Fundy Z and Fundy Z is running in a different folder on a different server and no one updates the backups to start backing up Fundy Z and two years go by and the backup reports are saying hundred percent everything backed up every day everything's great and then they have a failure of the Fundy Z server and they say oh great let's go restore those Fundy Z backups and then the heart palpitation start as the person realizes as they look through the backup logs to find the Fundy Z stuff that it's not there because it never got selected to be backed up in the first place that is a mistake I see way too frequently in the world and I again an annual review of your backup selections and just looking at you know what's changed and then also having IT processes that make sure that part of installing a new system is thinking about how that system is going to be backed up all really good things how confident are you that your nonprofit could restore a backup within a few hours so if you are the IT person and your executive director came to you and said hey I need this file from yesterday can you please restore it you're talking five minutes talking ten minutes you're talking you're getting down to knees and praying if you're the executive director you're not the IT person what's your level confidence if you go to the IT person that they would be able to pull that file so we're seeing a lot of aids here I've seen only aids so far it's great people are very confident in their backup that's great glad to see that not a ton of responses here but those who came in all aids still all aids come in excellent excellent excellent people have good backup systems in place very happy to hear that other planning considerations and we're we're home stretch now workday versus weekday how we respond be different if people are in the office versus at home if you're distributed workforce like a deal where a round table it's a super easy question it's kind of the same all the time but that's some something just to think about real quickly for your organization and see work through it you know think about training and practice again I don't encourage organizations to run again unless you really unless you're like a four or five I'm at you know your services really need to run the event of emergency I don't necessarily recommend running actual fire drills of your entire disaster response plan but again actual rigorous scenario planning you know actually just talk it through think it through with the key players what would happen abc is absolutely worth it for every type of organization from the you know that semester bound shelter all the way to the community theater looking at your insurance again just like with vendors it's not the time to figure out you don't know who your internet vendor is again you know right after you had a flood is not the best time to find out that your insurance doesn't cover floods or doesn't cover water damage from fires or ridiculous things like that cyber security insurance I get asked about a lot certainly worth looking into for your organization but a lot of times it's redundant with other coverage you already have or with coverage you already have from your outsourced IT providers so for example a lot of people have fixed rate managed support with roundtable and then they are going to pay for cyber security that's going to cover the labor for someone to help them recover from like a ransomware attack or something like that and we explained them when your labor is covered because you're already paying us the fixed rate for whatever supports if you had a ransomware attack we're going to do all of our labor is already covered under that you know as long as you have that contract with us on the other hand if they were paying hourly then we would say yes you might want to consider that cyber security assurance because if you needed 20 hours of us unexpectedly that could be a big bill that you wouldn't be anticipating so understanding where you're already covered and where you're not is a good good thing to go through having written down putting it on paper putting it in a document there's all kinds of good templates for this and idealware is going to be I believe releasing a article on this that will have some templates in it in a couple of weeks so you can be on the lookout for that again if it's on the server that's unavailable that's not going to help most people don't make this mistake but it's a reasonable thing I'll give you guys a quick tip of something that that I've done which is a lot of times like administrative passwords to certain critical services become a stumbling block if you know your IT director isn't available or something like that I would on an annual basis we had two executives that were sort of the backups for IT and we had a set of about five credentials like our Windows Active Directory domain admin the administrative credentials for our website a couple other things just that would allow you know allow them if I was completely gone or unavailable and my IT colleagues were unavailable and they really needed to hire somebody and needed to be able to give them access to the systems it was those key credentials that would get them in the stuff I would put those on a business card and I would give it to them once a year and I would tell them to keep that where they keep their credit cards and their other things and you know that was that that way if their wallet or business got stolen they could tell me that and I would have all of those things and know which credentials I needed to them go change quickly and then once a year I would just update all those ask them for the old business card shred it give them a new one really easy took 15 minutes a year and then they have those if in the event that something happens I was like a really easy solution I forget where I found that but I thought that was a smart thing in review oh and by the way the point I was going to make I forget about that if you if you put your whole disaster thing as a cloud document if you put up in one drive or Dropbox or something like that this is not something that usually if it doesn't itself have any credentials in it it's not something that needs to be super highly secure and you know I'm not saying publish it where you know Google's going to find it but you can certainly put it in a document where all you need is a link and then you can create a short URL for it and then you can put that on that little business card so now your executives and all the people in the plan actually have a business card that has the credentials for key systems if they need those and a link to the disaster plan that if they can get to any machine with internet access they can type that URL in and there's the disaster plan that's pretty cool and that's just on a paper business card again really easy to do interview don't be afraid to imagine these bad scenarios don't don't get you know bummed out or anything it's just you know these things are going to happen right they have happened historically they will happen again hopefully they're not going to happen to any of us but they do happen and the better prepared you are for it the better you're going to be able to serve your constituents and your staff and the less stressed you're going to be in that moment all right review all of your systems make sure everybody understands safety first and and write it all out and with that I think we are we are in wrap up a couple more links there's some links here for you for emergency preparedness for workplace plans there's some templates there they can get started that kind of came out formatted a little bit strangely and with that we'll do QRA and don't forget to fill out our survey and there's no link there all right Brian do you have a survey that you're going to send people to otherwise we'll uh yeah since we moved to go to webinar the survey will as you close down it will pop up and then we'll also email it to people within 24 hours through the system so really and we finished 10 minutes early so we got time for Q&A I'm going to go through and to see it looks like all the the things I'm seeing are answered to questions so far but I'm just going to leave all these out in case new questions come in anyone has a question by all means just chime in if you have a comment or any tip on disaster planning that that you want to share oh someone's playing with the mouse cursor on screen I'm sorry I was not getting out there it's a security ninja webinar coming again since it is over we're not planning to rerun it as a live series but as Brian mentioned for the ninja webinars all of the recordings of every single webinar we did including Brian's which recession five or six forget which one it was which was on digital privacy all of those are available for free all of the slide decks are available for free and you can get all of those at ninja.rtc.nyc and then answer that question apologies for the cursor sitting on the screen when I was leaving that I wasn't saying anything I'm like paced around the room so I just look at the slide very briefly so I apologize if that was driving people nuts a question from Kelly we contract out our IT support is it reasonable to ask them to or expect them to do a test backup occasionally the short answer to that Kelly is 100 yes the longer answer is if you have a fixed rate plan with them if you're paying some fixed amount per month for a set of services then honestly you would need to review your contract to see if tests of the backup system are included in that or not and you know you could ask what they would charge to do that on a you know some basis if you pay them by the hour then they're not going to be doing that on their own because they'll make the assumption I you know that you won't want to pay for that so you'll have to be proactive about saying we'd like to pay you to test our backups what would you charge us to you know do some random restores and verify their backups are working but in most managed service contracts tests of backups are included in roundtables that is included depending on the service level we either do it monthly or quarterly for different organizations and we actually do go ahead and restore files and some of the newer systems we use a high availability system called Datto and Datto now actually does its own self-restores every day which is really cool so Datto the system where it creates like these virtual instances of the systems you're backing up and so it literally just within its own environment it mounts the virtual system and restores a file from the previous day and then tells you send you a notification saying we did this and it was successful it's just super cool and from a cost perspective I'd highly recommend that when you go to renew contracts with particular vendors you talk to them about adding that type of testing as part of it because then they have the motivation to keep you as a contract at that point and it's much easier to add it in then and I would say from touching on the cyber security piece I would also Kelly if you're going to have a conversation about them there's a whole set of things that optimally would be covered under a service contract and and I would also you know every month we're doing test backups for organizations we're patching all their systems firewalls desktops servers everything with with current firmware or software and that would include third-party applications like adobe and java and things like that and then also testing the credentials that we have for their systems to verify that they work so that we know we can log into their godaddy account to their active directory and things like that and that's a you know I would consider to be a sort of best practice for any managed service provider to to be doing that thing all right any other questions I think we're good my brand I'm happy to stick around for five minutes but it looks like we're clear of questions excellent thank you thank you so much for coming out and talking to us today we greatly appreciate it