Okay. Hello, everybody. Welcome back to the forum webinar series. If you don't know forum, we are the professional membership program of the National Trust for Historic Preservation. This webinar series is made possible by members of the Preservation Leadership Forum, and we sincerely thank those of you who are on with us today. My name is Michelle Pryor. I'm the coordinator of publications and programs in our Preservation Resources Department, and I run this webinar series. For those of you who have attended our webinars in the past, I have taken over for Rebecca Weist, and it's nice to virtually meet all of you today. Before we begin, I have a few technical things. You can download handouts from today's webinar directly from the header marked Handouts on your control panel. There you can find a PDF of today's slides. As for what to expect on today's webinar, first Diana Maxwell, our grants manager at the National Trust, will share what grants are available, what deadlines are coming up, and tips for your application. We will then move to questions from the audience. You should see an option to send questions to staff on your control panel. If you have questions during the webinar, you are welcome to submit them, but we wait until the end to answer all questions. And with that, I would like to introduce today's speaker, Diana Maxwell, our grants manager at the National Trust. Thank you for being with us, Diana. Thanks, Michelle. Hello, everybody who's either listening live or listening to this in the future with the recording. I always like to think that it's nice to put a face to the name, so that's what I like to start this presentation with. If you've emailed or called our office in the last, oh, six months or so, you likely talked to myself or Lizzie Barringer, who's the coordinator for grants and awards in our department, and that's who we are, that's what we look like. So we are located in Washington, DC in the Watergate office building, and that's our contact information up on the screen, the email and phone number, and then the website as well. So today I'm just going to talk about a few different things. First, why you should care about the forum. Then talk about our National Trust Preservation Fund and then our special funds, our grant application process, and that kind of goes for all of them. And then I'm going to talk about the African American Cultural Heritage Action Fund briefly. Some of you may have a big interest in and then the National Fund for Sacred Places, which is another one of our, these two are separate. They have sort of a different way that you apply and a different, they work differently than our other grants, so it's good to talk about them separately. And then finally we'll have some time for questions at the end. So first, why forum?  Forum is a membership level of the National Trust. We have about 2,02100 members. And basically a place for professionals interested in preservation to sort of be together. We offer networking and interactive sessions, things like this webinar. And it's also probably most importantly, you have to be a forum member at the organizational level to be eligible for the grant funds. So this is $250 a year. It's something that your organization needs to hold. And if someone in your organization has an individual forum membership, that can be upgraded to the organizational level membership by speaking with someone in our membership office. So that's just sort of the nice thing to get started. And forum membership also comes with discounts for trainings for the conference, things like that. So we'll kick it off with the National Trust Preservation Fund. This is our largest grant program. We have kind of, when we talk about National Trust funds, this is sort of what we think of as our biggest program. Over a million dollars is available nationwide each year. You do need to be a nonprofit or government agency to be eligible for this grant and really for any of our grants. These are grants that are for planning preservation projects and for educational, planning educational activities. So they're small grants. The idea is that it's a little bit of seed money to get you started. We found that it really can help an organization get money from other sources by having that National Trust kind of background there. They require a one-to-one cash match. This match can come from a foundation, an individual donor, someone funds that are on hand, but it does need to be cash. It can't be an in-kind donation or something like that. This grant fund has three deadlines a year, February 1st, June 1st, and October 1st. And so that's coming up pretty quickly February 1st.  So for all of our grants, you apply through our online web portal, which if you go to our forum website, you'll be able to find a link to that at the bottom of the grant pages. And you'll sign up there as a program called Foundant. And we found it's pretty easy to work through and it's a good program for us and it makes it easy for us to see your applications and you can upload photos and things like that. So everything is done through that system. We don't take anything as a paper application. So talking about sort of what these kind of projects could be for education and outreach, these are preservation education programs or activities aimed at the public, particularly programs aimed at reaching new audiences. So innovative techniques and formats, conference sessions. The way we fund conferences is typically through paying for speaker fees. We can't do a lot of other types of conference support, but typically through speaker fees is the way we do it. And this example here was a tour that you sort of took folks through. Thank you. Through Los Angeles, it went to the UCLA IM Lab for the IM Lab Interpretive Walk. It was a monthly somatic walk going through downtown Los Angeles and a public website too. So participants would be given a packet of cards featuring historic kind of photos of an area. You can see that black and white in the center and then go through the exact same location. So you could learn about the history of that site and also be in that site presently. For planning projects, this is support for professional expertise, for hiring consultants primarily. Architectural consultants, engineering, archeology, planning, law. You know, you can see some of those kind of types of things up on the screen. An example of this, we gave a few years back with the Chisholm Trail Heritage Museum in Texas where they applied for a grant for a historic structures report for a school from the 1870s. So that's sort of the, you know, the meat and potatoes of our grant program are these sort of smaller planning grants that are really useful and can be really effective in getting a project started. We also have some limited emergency and intervention funding. This is awarded for emergency situations. Again, nonprofit organizations and public agencies. This can range from $1,000 to $5,000. And the important thing here is that no cash match is required. We understand, you know, if it's an emergency situation that it's likely you don't have a match on hand. An example of this would be a grant we gave to the Georgia Trust for Historic Preservation where this building gains hall at Morris Brown College had burned. And the Intervention Fund after the Georgia Trust came to us after the fire and we were able to help them engage a structural engineer who designed a bracing system to support the original exterior walls as well as the new interior, since the existing interiors weren't salvageable and a roofing system. And this building is just a few blocks away from the new Atlanta or newish, I guess now, Atlanta Falcon Stadium. And the economic sort of growth that's expected in that neighborhood bodes well for adaptive reuse of this building. So we don't have a lot of information on emergency funding on our website kind of on purpose. What we want you to do is to email us or call us if you think your project qualifies. We have sort of, the funding is quite limited, so it's helpful for us to be able to talk to you about your project and what you require before, you know, kind of making a decision there. We don't want you to put in an application for something that's not going to be eligible. So basically the first step for this is to send us an email, let us know the situation, what you're looking for, and we'll get back to you with, there's an access code so that you're able to submit the application. So that is emergency funding. And then we do have several funds with special exceptions that kind of are tied into the preservation fund as well. So all of these funds are part that you see on the screen are part of the National Trust Preservation Fund. You'd apply to any of these funds throughout the regular process. So you wouldn't be looking for specifically for the battlefield fund as a separate application, you're just applying to the preservation fund. But these funds are located, more information on all of these is located on our website. Generally, they're sort of geographically specific in like the Colorado and Hawaii funds. The battlefield fund is focused on funding battlefields. That's pretty straightforward, I think. And then the Deutsch Memorial Fund for Rosenwald Schools is focused on Rosenwald Schools. Like I said, more information on our website. If you fit into one of those categories, I suggest you check that out. And again, you would just be, these have the same deadlines and the same application as the preservation fund grant. So they would all be due the next deadline of, excuse me, February 1st. So that was our National Trust Preservation Fund. And then now this next section focuses on some of the other special funds that have their own applications. The first, and actually I lied because the Southwest Intervention Fund no longer has its own application. But this fund is geographically restricted to Arizona, Colorado, New Mexico, and West Texas in Utah. And it's a very specific definition of West Texas, which is on the website. You're interested in this, check that out. Again, preservation planning efforts and responses to emergency threats. So it's very similar to the preservation fund grant. You will still apply to the preservation fund through that application. Local partners, nonprofit organizations, the government agencies, the difference with this is that they can go up to $10,000, but otherwise similar restrictions. So an example of one of these we gave in New Mexico used grant funds to stabilize an archeological site that had well-preserved petroglyphs and a midden, but the masonry dwelling was in need of stabilization. So that's the kind of work that this fund can do. It does need to have a connection to traditional cultures. So if you have questions about that, again, send us an email, give us a call. We're happy to talk through that with you. So now the Joanna Favreau Fund for Historic Preservation, this does have a separate application, a separate grant deadline. This is for planning, education, and brick and mortar grants. This is one of the few funds we have that does allow for physical preservation work. And the amount that you can spend is a little bit higher. It goes up to $15,000. Still a one-to-one cash match, still nonprofits and government agencies only, and the deadline for this is March 1st. So this application will be up probably just after the first of the year, once we're back in the office after the holiday, and you can see that application there. This only started last year allowing for brick and mortar grants, and it is a national fund, so it is pretty competitive. I would say more competitive than some of our other grants, but it's a good opportunity to get that brick and mortar money. It is one of our donor-advised funds, which means that the process for choosing those grants takes a little bit longer. So the grant deadline is March 1st. You won't be notified of the results of the grant until July 1st, so it's a little bit of a longer lead time there. An example of this one, the photo on the screen is from Notre Dame of Maryland University in Baltimore, and they used a FABRO grant to hire a preservation consultant to create a preservation plan for this building, Gibbons Hall, which housed the first Catholic College for Women in the U.S. to award a four-year degree. So that's the FABRO funds. Along with the FABRO funds is the Cynthia Wood Mitchell Fund for Historic Interiors, and this is similar to the FABRO funds in that it can do planning, education, and brick and mortar grants, but this is limited to the interior of a historic building, so this is focused on historic interiors. Again, it can go up to $15,000. The math requirements are the same sort of applicants, same deadline, March 1st. They, again, and also as competitive as the FABRO fund is. So if you have questions about these, again, always happy to answer them. People often ask, can I apply for both? You can apply for both in a round. However, you won't receive both grants. We can only give one round, one grant per round. So I would suggest that you, you know, apply for, put the application in for the one that's most pressing to your organization and maybe hold the next one until the following year. So the project here that was funded by a Mitchell Fund grant is at Texas A&M University, and they received a $10,000 grant to prepare a preservation plan for the decorative painting treatment of Schott's Hall, which is, you can see those murals at the photo of that screen. Another one that we funded was in Charleston, where the grant helps the Historic Charleston Foundation evaluate the plaster within the Aiken Ret House Museum's kitchen house using a combination of non-destructive infrared thermography and traditional sounding methods. So we have some, you know, interesting projects that come up through this, but again, it is limited to historic interiors. Another fund that is a little bit different from the preservation funds is the Hart Family Fund for Small Towns. In some ways, this is closer to the preservation funds than Fabro and Mitchell in that it is limited to planning education grants. It's limited to projects in a, with a small town sort of population that means, for us, that means less than 10,000 people. This fund can go up to $15,000, match requirements are the same, and the deadline for this fund is May 1st. So we typically try to have applications open eight weeks before a deadline, so this will be coming up in another couple of months. An example here is a town in California which supported a feasibility study for the reuse of a former Carnegie Library that was in good condition but just wasn't used anymore and the community wanted to kind of figure out what to do with this next. This next fund moves away from sort of typical building grants. The Peter Brink Leadership Fund is for mentoring, basically, and peer-to-peer and networking between preservation organizations. So what this does is reimburse travel costs and honorariums for the mentor up to $2,500. This is a rolling application that is accepted throughout the year. And the example here was for Preserve Grand Barrier in Texas and they had the Esther Hall, who was Chair of the National Alliance of Preservation Commission, travel there to mentor this group in leadership, training, and strategic planning. So they could sort of work in this, in this case it wasn't a one-on-one, it was sort of a one-on-small group, but we see that and that works as well. So if you have questions about this one, this is an application that you do need an access code for. The grant office can give you that access code, send us an email or give us a call and we can, you know, just to talk through your project, let us know who you want to be mentored by or work with and then we can go from there. But this is a rolling application so there isn't a set deadline for this. We have some other special funds as well that are geographically restricted. The Appell Fund is limited to Central Pennsylvania. The Bardas True Fund is limited to the Eastern Shore of Maryland. And the Henry Jordan Fund is Mid-Atlantic States, which are listed on the website. This is, this sort of focuses for the protect, the Jordan Fund focuses on the protection of natural and cultural resources and has a deadline of June 15th, June 17th this year because the 15th is on a weekend. Bardas True does allow for brick and mortar work and it has one deadline each year, June 15th. The Appell Fund also allows for brick and mortar but is limited to specific counties in Central Pennsylvania and that deadline is March 15th. So for all of these, if you are in that geographic region, I suggest you go on the website and see if that is a grant for you. Generally similar guidelines to our other funds and restrictions although two of these three do allow for brick and mortar work. So that's sort of quick and dirty on the majority of our funds. I'm going to talk about the grant process for all of these funds and then we'll come back and talk about a couple of the other larger grant programs that we have that don't tie in as well to the rest of these. So how do you apply? You go online to forum.savingplaces.org slash funding, go through the guidelines, look at the criteria, look up what deadlines are coming up and then at the bottom of that page, you'll find a link to Foundant which is our online grant portal. Everything has to be submitted through this portal. Things that you will need include letters of endorsement, your tax event status, a number of other things you can see on the screen and you'll be able to see in the application. It's not a super lengthy application. We try to only ask the questions that we need to know the information to be able to make the decisions. But if you have questions about it, again, contact us. It's generally a pretty good system to work with, I think, both for us and for our applicants. The review process looks like this. First, the application is checked in by grants office staff for basic eligibility and at this point we may follow up with you for clarification or for additional information. And then the applications are reviewed by the grant staff, by national trust field staff or other staff, sometimes by partner organizations depending on the grant fund. And then the decisions are made by a grant committee, which could be the grant staff and our vice president of field services or a grant jury, which could include national trust staff, national trust donors, or an external juror. So this depends on the grant fund, but generally that's the process. And then finally the applicant is notified, typically about eight weeks from the due date. Though occasionally this will take longer if we, for various reasons, we'll try to let you know what that's going to happen. All of the notifications come out via email. So it's important that when you apply at the top of the application page, it'll give you a couple of websites or a couple of email addresses, rather, to add to your faith list on your email account. We really encourage you to do this because our emails that go through the grant system occasionally get tagged as spam. And we don't know for sure if that happened, only you know. But if you don't get that email, you don't know what the answer to your grant inquiry was. So add those email addresses to your faith list to be sure that your emails are coming through because that is how we notify we don't send anything out as a paper copy. So, you know, we go through this whole process. We'll notify if you got a grant. If unfortunately you didn't get a grant, why? This is usually one of the questions that we're often asked when people contact us. And there's a number of reasons, but these are sort of the main four. The most common reason why you didn't receive a grant is because our funding is limited. We technically have nationwide coverage, but our grant funding comes from geographically restricted endowed funds. So the different locations, depending on what state you're in, where you are, have different amounts of money and can be more or less competitive based on that. The second reason is that there weren't the correct amount of matching funds. So you have to have that one-to-one cash match. If you don't have a match at all, we can't fund your project. If you don't have enough of a match, we may fund a smaller portion of your project. And this sometimes happens for that first reason as well. You may get a smaller grant than you requested because we didn't have the funds to fund your full project, but we still wanted to give you a partially funded grant. So those are the two top reasons. The third reason is that your project doesn't meet our guidelines or you don't have enough eligible expenses. So our grant pages list all of the ineligible expenses that we cannot pay for, and that's kind of a big reason why you get a smaller grant or not a grant at all. And something that's not always clear to people is that eligible expenses portion of it. So for example, you have to have enough of them that both our grant and the match to our grant would cover the eligible expenses. So if you have a $10,000 project and you're asking us for a $5,000 grant, that's great. You have $5,000 in matching funds that you've gotten from a local foundation, hooray. So then you have the cash match. That's great. You're going forward just right. But if $5,000 of your project is too higher in architect to create drawings for you and $5,000 is for the contractor to do the work for the building and you were applying to the preservation fund, you would only be eligible for a $2,500 grant because only $5,000 of your expenses were eligible. So that's where people often get a little tripped up, and that's one of the reasons why you may not have received a grant or you may not have received full funding. And then finally, your project may just not have been the right fit. If it's not preservation focused, if it sort of just has a general history bent, we're less likely to fund it or not going to fund it. We see this more with education programs than with actual building related projects just because of the nature of the project. But it also, you know, might not be the right fit.  If you're working on an educational project, but the photos you send us show that the roof is about to cave in, we may wonder why you are not working on that portion of the project rather than the one you applied for. So if there is something like that where perhaps you're working on multiple projects at once, then you maybe have funding for another project through other sources but are applying for another portion. It may be, it's good to mention that to us as well, just so that we aren't confused about why you're applying. So again, you know, reach out to us before you apply. We can talk through your projects to see if it makes sense for our funds. Sometimes the timing is just off. We've, you know, we may have spent all of our funding for the year. Our fiscal year cycle runs July 1st to June 30th. So the February round is right in the middle of that. The October round is our first round of the year. So by June, by the June round, sometimes we are out of money in certain areas. Other times that means we want to spend the money before the fiscal year is up. So it can go either way but, you know, we encourage you to come back with another application if you don't, you know, don't get selected the first time. And also consider one of our special funds. We may recommend this to you or we may just consider that ourselves and transfer the application over. Usually you need to do that yourself but, you know, we're happy to do that if that makes sense for you. So that was a lot of information about many of our grant funds. But now I want to just talk quickly about two of our newest grant programs. The first being the African American Cultural Heritage Action Fund grants, which we just had our first round of last year. And we're right getting ramped up to have our second year of these grants. So this is separated from the rest of the grants in this presentation because it's really a different program. The National Trust for Historic Preservation's African American Cultural Action Fund grants are designed to advance ongoing preservation activities for historic places such as sites, museums, landscapes, but places that represent African American cultural heritage. And the fund itself is different from our other grants in that it has a two-step application process. It starts with a letter of intent or LOI phase. So it begins with this LOI where folks apply to this and from those applications we narrow it down and invite a much smaller group to submit a full application. We have four categories that applications can be received in, capital projects, project planning, organizational capacity building, and program and interpretation. And this is, like I said, only the second year we've had this grant fund and we expected to be very competitive. It was very competitive last year. Last year we received 830 letters of intent and we invited 50 organizations to submit applications and then we were ultimately able to grant to 16 organizations. So these are larger grants, 50,000 to $150,000, and we welcome your letter of intent. Again, we can talk about this too, but on the phone, by email, I'd also encourage you to go to this website here that's listed on the screen or just Google African American Cultural Heritage Action Fund grants and you'll get to our site to see sort of the 16 projects that we chose last year that can be helpful in helping you sort of see what we were looking for and what we were interested in. The letter of intent is due January 15th. This process is open now for those letters of intent. We plan to notify everyone by March 25th if they were selected to move on to the application phase or not. The applications are due May 1st and this isn't on here because I think I ran out of room, but notifications on the applications will come out late June or early July. So that's the action fund as I shortened it to in my brain because African American Cultural Heritage Action Fund is a mouthful. Another fund we have is actually a fund we share with a partner organization and this is the National Fund for Sacred Places. This provides capital funding for houses of worship and it's a partnership between the National Trust and Partners for Sacred Places who are great non-profit groups based out of Philadelphia. This was meant to grant, it's granting a total of $10 million over four years to approximately 50 houses of worship and we're in our third round of this funding, I think. It sort of blurs together at a point, but congregations must meet certain criteria and they receive training and support from the National Trust and Partners for Sacred Places. And then the opportunity to apply for a capital grant ranging ranging from $50,000 to $250,000. So there are a lot of guidelines and criteria for this fund. I very much encourage you to go to that website and this fund for sacredplaces.org, read over that information. It's really helpful. There's a lot of great sort of tools to walk you through your project and your congregation. The LOI is due in April and that should be opening up relatively soon. And then again, this is sort of that two-step process, the letter of intent, the LOI, and then where we narrow the applicants down and then invite a smaller group back to submit an application. And then final notifications for this are due in October. And again, that website lists all of the projects or the congregations we have funded to in our previous cohorts so you can see there sort of what we're interested in and what we're looking for. We take ultimately about 15 congregations a year for this fund. So it's a good reminder, you know, our ultimate goal is not to be Uncle Scrooge and kind of hold all of our grant funds in a vault somewhere. We want to give this money away. So, you know, we want to fund your project. We want to help preservation around the country and help people who are doing preservation. So we want to give this money away. So please help us do that by, you know, putting in your applications and emailing us to ask about your projects and reading the guidelines and, you know, going through that. So we have time for questions now. You can use the chat box. I think, Michelle, you were going to talk more about that. Yes. Yes. Thank you, Diana. Thank you so much for all this information. And as you can tell, Diana is our grant guru. And so we're going to start the question and answer portion of the webinar. So again, if you have questions, feel free to type them in on your control bar there. We'll answer as many as we can. So we've already had a couple coming in while Diana was speaking. So thank you everyone for your patience. And so let's answer a couple of these now. And again, if we don't get to your questions, don't panic. We will post a blog on our forum or blog post on our forum blog answering all of these questions. So let's start with Diana. Can you receive funding for same projects from both the African American Cultural Heritage Action Fund and the National Trust for Start Preservation Fund? Yes, you can. Since there's separate applications and decided separately, you can use that. What you can't do is match the National Trust funding with National Trust funding. So you would need to have a different match for your National Trust Preservation Fund, different money on hand. Okay. Next question. Does the African American Cultural Heritage Action Fund cover excavation projects? I think I would need to know more about that project to answer that question just because it doesn't cover new construction. If it was archeological excavation, yes. But we would need to talk about that more, I think, offline. So if you can send us an email, we'd be happy to answer that. Great. Thank you. Next question. How do you get reviewer feedback from the grants committee? Sure. So you email grants at savingplaces.org. And then we will go and see sort of what reviews said and let you know what we hear, what we knew. Wonderful. All right. Next question. For the Cynthia Woods Mitchell, is the match financial or are you able to use in-kind as well as financial? It has to be just money. It can't be an in-kind match. Okay. And you don't, for any of these, you don't have to have the match on hand when you apply. But you do need to have it by the time you need to pay your contractor at the end. Okay. Great. Next question. On to the grant limits, even with matching funds, fall short of the kinds of fees that I see for the studies you've noted as examples. HSRs and our nominations, condition assessments, et cetera, often cost in excess of $30,000. Are there exceptions to the funding limits? Do you foresee these limits increasing anytime soon? Let's start with that. Yeah. I think that this is the grant program that goes back to about 1969. And not a lot has changed really since then, except for the cost of things. So this is something we're actively working on to try to up those limits. It's going to take a little bit of time. We've tried to sort of, we've been able to bump up some, like the Fabro Fund and the Mitchell Fund working with our donors to bump that amount up. Some of them, it's really hard for us because we don't have a lot of money in some of these grant funds. So it's difficult for us to give a $10,000 grant for a fund that only gives us $3,000 a year. Generally, there are not, you know, we have to stick pretty close to those guidelines. But we are sort of working on this to try to see if there's a way that we can use these better to help better. And, you know, a lot of times we're only one piece of that puzzle in terms of the funding. All right. Next question coming in. If a grant application is turned down, is it possible to resubmit essentially the same grant but with improvements? Yes. You're going to need to fill out the application again. We can't just look at the same application. But you can, as within our program, you have access to everything that's been submitted from your organization. So you can go back and you could copy and paste over things that, you know, were either. Think something like the history of the site probably hasn't changed since she submitted it last, things like that. You could copy and paste over. Let's see. We're having a lot of great feedback on your presentation, Diana, looking for the next question. Let's ask this one. Tell us a little bit more about the letter of intent. Should the letter show the history, the needs for facilities, and etc.? Yeah. So the letter of intent is actually, like I said, it's an online form. So it has questions that you need to answer. So it will ask sort of, you know, what is the, what is the history of your site? What project are you looking to do for the, I'm assuming you're asking about the Action Fund. You may be asking about the Sega Places Fund, in which case Sega Places Fund will ask about your congregation as well. But all of those questions are there. It's not something where you're writing a letter to us. It's, you're filling out this form and attaching the documents that we're requesting as well. Great. And I just want to say to a couple of our listeners that if you're looking for a copy of the PowerPoint on your control bar, you should see an option that says Handouts. That should be on the right side of your screen if you haven't moved it. Again, you'll see Handouts. And if you click the little arrow, there's a National Trust Grants PDF for you to download. And that's all the slides Diana was using today. So no worries. We'll get that to you. Let's see. How about this one? Can we apply for the African-American Cultural Heritage Action Fund grant and one for something else from the HPS? From the preservation fund, yes. You can apply for two projects. We can grant up to three grants in a two-year period for any organization. So that could be any of our funds or any of our grants. Great. Here's another one. If an organization focuses on historical places focused on black history in Oregon, is that considered a landscape? Last year's funding focused on buildings in one historic district. So this sounds like somebody's gone online and looked at those prior projects, Diana. Yeah. I think that would be a landscape. It would be an area. I think it would... It's still a place. You know, we try to give as many examples as possible of what we mean by place. We can't get all of them in there. But I think that that would be included. Yeah. Next question. Does the NTHP provide any links to recommend consultants for things like an HSR? Or do we have to do that research on our own? Yeah, we don't. I think once upon a time, before my time here, we did, I think a lot of times we don't necessarily know who's in your area. So we would encourage you to go to either your State Historic Preservation Office or probably your statewide preservation organization to get more examples of local folks. I personally don't have that information. So we would recommend that you reach out to somebody local closer to you. Okay. Here's a question. Do you no longer have the Gwen Turner Fund as an active funding program? So that's included in the National Trust Preservation Fund. A little bit of history, there were... The way that the grant fund used to be run, they were kind of called... Many funds were called out specifically in different areas. The Gwen Turner Fund is in Alabama. And that is just included in the National Trust Preservation Fund. So that's the application you would submit. Okay. Next question. Are there templates or strong examples that we can use when drafting letters of intent? We don't release that information. We don't have that available. I think that everybody's project is so different that it wouldn't really be a good example. I think that would be a hard kind of apples to apples comparison. I think you just kind of have to do the best you can, be as clear as you can. Remember that we may not know what your... You may be very familiar with your project, but we aren't necessarily familiar. So it's helpful to give us sort of that background of what it is and what you're trying to do and just be as clear as you can. And in the letters of intent this year, one of the things we're asking is sort of how far in the planning process you are. We found last year that we weren't sure where people's projects were. If it's just a grand idea, is it something that you talk to consultants and have budgets for and things like that. So we've added that question to the letter of intent this year as well to see where you are in that planning process of doing this project. All right. Great. Next question coming in. Is national trust funding only available for buildings? Or can projects that preserve, say, historic vessels listed on the national register of historic vessels apply for any of these funds? Yep, vessels could apply. We've given grants to ships in the past to train cars. It's easier just to say places, but transportation vehicles, I guess, can be included as well. All right. I noticed the previous ED did not complete the follow-up info from a previous grant received. Will I be able to submit a new request or will I have to figure out the outcome info from the previous grant received? That's a good question. If you have any sort of send us an email, basically, with your organization, but generally if you're coming into a situation where the person is no longer there, you have no knowledge of the project, we can work with you to try to just get some sort of basis of what was done. Sometimes, depending on your project, there may be money, additional funding tied to a final report. So that's why I'd encourage you to email us so that we can make sure that you're getting the full grant amount. But if it is just one of our preservation fund grants, we do like to have everything closed out before, so that you're up to date and in good standing, your organization is. But we understand that you may be coming into something where you have limited knowledge of the project. So if we can just kind of work through that with you and get it taken care of, just send us an email. Great. Next question. Do sacred places include injured petroglyph sites and other such sacred sites? They generally do not. Send us an email and let's talk about your project. The funding that we received for this grant has limited it to Christian congregations, and we have a little bit of more money that can be used for non-Christian congregations. So send us an email. Let's talk through your project, and we can go from there. All right. Next question. If we're applying for a project in progress, can we use the architect already involved, or will we have to get quotes to start with a new firm? The cost won't reach $50,000. Generally no. We do need to sign off on your consultants before we can send you a grant agreement. So, you know, we'd like that you sort of have done your due diligence in terms of making sure that you have, you know, bids that are reasonable or what you need to do, but we understand that, you know, some places you may need, there may only be one person who can do this work. And we can grant, you know, we're okay with that. We want to talk about the situation. So put that person's name on your application. Most likely it'll be fine. Hopefully you're using someone who has preservation experience, and, you know, we have no problem approving them. Okay. Next question. Are there priorities within the African American Cultural Heritage Action Fund for addressing particular aspects of African American cultural heritage? Not quite yet. That may be more information that comes out. I think right now that there are not. Here is a follow-up to an earlier question. If you apply for a transportation vehicle, does it have to be listed on the National Register to be eligible for a grant from the National Trust? It doesn't. Nothing, your, any project does not have to be designated. It's helpful if they are. It sort of gives you a starting point, especially for the historic significance of the project. Just having to, you know, having that information on hand helps you there. But nothing, building, ship, you know, train car, landscape does not, they don't have to have any kind of designation to be eligible for the fund, any of our funds. Okay. Again, for the African American Cultural Heritage Action Fund, we'll grant funding to help historic African American neighborhoods doing minor rehabilitation projects. There are historic houses count. So in lieu of one building, can we help multiple historic houses? Yeah, we could do that. And it would sort of be, you'd be applying, I think, for a community, for a street, something like that, that would be eligible. I think one of our projects from last year, it wasn't a capital project, but it was an interpretive project in Nebraska was focused on different black home setting sites. So that's sort of the type of thing where you're looking at a number of places that were not, you know, all in the same geographic place other than being in Nebraska. But with a project we funded. Awesome. Thank you. Another question. Can a separate non-religious NPO, whose mission is to preserve and operate a church building, apply for a sacred spaces grant on behalf of a congregation slash building, or can only the congregation apply? So as long as the preservation organization is related to the congregation, yes. So we see a number of congregations where they have a friends group, generally, that works on the preservation of the building and raises money for them. In that case, yes. But the church still needs to have an active congregation related to it. It still needs to be an active church. Okay, let's see. A lot of great questions here, everybody. Thank you for staying so engaged. Here's another one, Diana, also with the Action Fund. Is there an average dollar request for the African American Cultural Heritage Action Fund? I think generally people asked for the largest amount that was available. We have some sort of sliding ranges that are on the website, depending on which category you're applying with. Those amounts are different. So I would look there, and I would say, we noticed last year, you know, a lot of people were applying for the maximum amount, that $150,000. And it was unclear if those numbers, again, I had mentioned earlier, you know, we were asking where folks were in the planning process, partially because it was unclear if all of that money was really needed or if it was just being asked for because it was the maximum amount. So the more specific you can be, I think, the more helpful it is to us. We wanted to, ended up wanting to fund more projects that, with the amount of funding we had available last year. So very few of them got the full ask that they had. So if you can be pretty specific about what you're asking for and the more you know about the real, actual cost of it as you're submitting these applications, that's really helpful to us. Great. All right. Let's see. Next one, could you describe the top three things you're looking for in the three letters of endorsement required for applications? I believe that's for the Action Fund. It's probably for any of them, the Preservation Fund's expertise as well. I don't know if I can give you the top three. I can say it's more helpful if it's from a person that actually knows your organization. You know, having sort of a generic letter from your congressional office doesn't really help us that much. But if you have a partner that knows your work or that knows the project really well, that's helpful. If it's somebody you've received a grant from before, that can be helpful as well in terms of kind of showing that your organization has done good work and, you know, can manage a grant well. Yeah, I think that that's the biggest one is it's sort of the more, the more the letter, the person writing the letter knows about you, I think is more useful to us than just sort of having a big name there. Next question. Here's a question. What grant would be best for the restoration and preservation of a currently empty African-American church building? That's a good question. If it does not, if the church building does not have a congregation actively attending it, it wouldn't be eligible for the Sacred Place Fund. So my first suggestion would be the African-American Cultural Heritage Preservation Fund. If it's in very early stages where you may need a feasibility study or a historic structures report, it may be the best place would be the National Trust Preservation Fund. Again, that's a lot smaller money, but could could get you started. I would say probably I would apply for the action fund and see where that comes down and then maybe also apply in the February or June round for the preservation fund, especially if the smaller preservation fund grants could build up on something for the action fund grant. Okay. We'll let Diana take a small breath here and I'll remind everyone once again on the right side of your screen where your toolbar should be that you've been using to attend the webinar today. You can find a PDF copy of Diana's PowerPoint. It's under handouts. If you click the small arrow, it will show you a downloadable PDF right there on the screen. If you're having trouble seeing this PDF, it will be added along with the recording to our website, which is forum.savingplaces.org. I will show that information at the end of the webinar, but we have a couple more minutes left for questions. So Diana, if you've taken a breath. Okay. Next question. Does the African American Cultural Heritage Fund require a match? It does not require a match. However, we like to see one. So if you can get a match, even if it's not a full match, to show that you have other interested parties and funders, that's great and that we will, and we say this in the guidelines, we do kind of not officially score them higher because there's a real way to score that, but they do get a boost in our eyes by having a match. All right. Next question. When a project focused on building organizational capacity to implement a comprehensive program aimed at preserving African American culture and community building, the eligible, especially if it isn't focused on an edifice? Possibly. I think I would want you to email us with more specifics on that. I don't want to give a yes or no answer without having more specifics, but send us an email. Next question. Would any of your funds be suitable for legal costs, say to obtain ownership of a long clothes and abandon African American church and the related cemetery? Possibly. The preservation fund could be used for this. Probably would be the best option. The only sticking point we have with legal funds is that our law department has to review them. They have to be able to fit in with our sort of internal guidelines there. So one of the things I know that we can't pay for are straight sort of lawyer's fees. The our department feels that it's not, that's something that, you know, you could go down a rabbit hole and just be spending thousands and thousands of dollars. And there's no sort of limit to that. So generally anything associated with legal fees, it's sort of a set cost, so it may be for something like hiring an expert witness or court filing fees or things like that that are sort of a set amount. But we can do that. And sometimes our emergency funds help with this as well. But the preservation fund would probably be your best bet there. Okay. Here's another question. Is it mandatory to have a photo, name and location of the project published to the public? Can the site remain anonymous? If it's something, I think we've run into this with some sacred sites, Native American sites. It doesn't have to be public. We need to see it. If it's something that needs to be anonymous, that can't be released publicly, though, you need to tell us that and be very clear about what can be shown and what cannot be shown in your application. Okay. And I believe that woman also followed up saying, specific to the action fund. Same for the action fund as well. Yeah, you just need to make sure, just know what that is and it's helpful for you to tell us why as well. Because generally we do, you know, you'll see on the last, or if you go to the website and see the project from last year, you know, there is a fair amount of marketing that goes with this when the funds, when the projects are announced. So that's going to be difficult to do if we don't have any kind of photo. So there might just be some work that has to happen around that. This is an easy one, Diana. A woman is asking, could you please repeat how many LOIs you received for the action fund last year and how many were invited to the full proposal? We received 830 letters of intent LOIs and we received, we invited 50, 50 back to submit full applications. And we have no idea how many are going to come in this year, but we do anticipate inviting a similar amount to submit full applications. Next question, are any funds available for flow-through grants? I'm not exactly sure what you mean about that, but if you mean sort of re-granting to other organizations, maybe. That's not something we do a lot of. I'm trying to think if there's examples of that. There might not be, but again, I think the, in the last couple of years, the majority of our funds have been such smaller amounts that they don't really make sense to do that. And our guidelines are such that that might not work depending on what the project was. If you have a specific project in mind, send us an email. Sometimes hypotheticals are hard for me to figure out unless I have specifics. So that would be helpful there. Okay. Now, Diana, we have time for one more quick question. And then we're going to wrap things up everybody. But again, these answers will be posted on our forum blog. So last question here. Do sacred places require a match? Sacred places do require a match. Definitely go on the website, check that out. Depending on the dollar amount, it's either a one-to-one or a two-to-one cash match. So that would mean if you're receiving a larger grant amount, then you have to have an even larger amount to match it. Lots more descriptions on that website. I would encourage you to go check that out. But yes, they do require a match. Okay, everybody. So... Reason up. Doesn't want to go. Okay. Oh, there we go. Yep. So thank you guys, everybody. Before I hand this back to Michelle, that's our contact information. You can also find that on our website. I would imagine if you Google National Trust Grants, you will be able to get to this information at some point too. Like I said, we're always happy to talk to people. Sorry. And send us an email. That's probably the easiest way to reach out the email. But Michelle, back to you. Back to me. Thank you, Diana. Well, everybody who's still on the line, there are some upcoming webinars if you enjoy today's content. So we have two schedules in January 2019. We have one of the 17th that will focus on Opportunity Zones and the Preservation Community. We also have one on January 30th that will focus on the Rosenwald School GIS project that was recently completed here at the Trust through a grant that we received through the National Park Service. So those should both be great. For more information on the upcoming webinars and to register, you can go to our website, forum.savingplaces.org slash forum webinar. And then to wrap up, I would like to thank Diana so much for taking the time to prepare today's webinar and being with us all today and for answering so many questions. We have more work for Diana as she'll answer the rest of the questions we weren't able to get to today. So again, thank you so much. All right, again, this webinar has been brought to you all by the Preservation Leadership Forum. You can learn more about us on our website. And that's all for today. Thanks, everybody. Happy holidays.


So what's up? My job is so stressful. I like my boss. The work is fine. Then what's the problem?  Hi, I'm Eileen and this is the lazy millennial. Today we're going to talk about how to deal with difficult people because I can guarantee you that at some point it's going to become a problem if it hasn't already. Here we go. Sometimes you're going to have to be around people you don't want to be around or associate with. Work, school, reunions, events, basically anywhere there are people. I think I should quit. What? No, it's not. That does pretty well. Okay, sit up. Give me the M&M's. All right. We're going to hash this out. We all have the freedom to associate and dissociate with anyone we want. Your co-workers don't have to be like your best friends. Brought it bowling. I didn't run it shoes. But the thing is unless you're prepared to quit your job and go live like a hermit, being a professional means you're going to have to learn to work with people you don't always like. Adulting. Sometimes when I'm struggling with people it helps me to realize whether or not they are truly toxic or just annoying with different interests. Here. Let me help you spot some key differences. Annoying. Toxic. So annoying. Uh, yeah. Toxic. Really annoying. Super toxic. Hilarious, but you don't want to work for that guy. The important thing is to realize the difference between the people who are actually harmful and the ones you're just not that interested in spending time with. With the actual baddies, the best thing you can do is to focus on being great at your job and not let them get in your head. I mean, unless it's like your boss or something and they're doing something illegal, in that case, don't just ignore it. But with everyone else, my advice is this. Be open and actually be compassionate. We benefit a ton from diversity. People see the world differently and that means that they might see something you don't or come up with a solution to a problem you wouldn't think of. Who knows? Maybe they can actually help you be better at your job. So guess what? What? I got a promotion. Oh my gosh. That's amazing. Thank you. Yeah. I really want to thank you for your advice the other day. I started avoiding manipulative guy at work and when I need a good laugh, I visit Carla in accounting. It's really cleared my head. That's really awesome. Part of living in a free society means learning to tolerate and coexist with tons of different kinds of people, but always remember you have a choice. It's up to you who you spend your time with. Use your power wisely. So Eileen, I've got another problem. Hey, thanks for watching the Lazy Millennial today. If you want to learn more about dealing with people at work, check out Fee's Essential Guide to Crushing It at Your First Job, which is available for free online at the link below. And if you want to see more videos like this one, subscribe. You know how this works. I shouldn't have to tell you.


Introductions. I am Marceni Chernoff. You've probably seen that I started as a co-organizer to this meetup. So together with Han Su, those of you that know Han, he unfortunately had to leave to get small kids. So Han and I, we're all looking for speakers for the next meetup. We also look for the next venue sponsor. So if your company wishes to invite this very honorable crowd and talk to us in a short manner for a few minutes, we're very happy to talk to you. I'm from Databricks. I started three months ago. We are actively hiring. So also if you're looking for additional challenges in AI and big data space, let me know. So I will introduce you to people and we can start talking about that. I am based in Singapore since 2011. I live in Sime. I'm a grassroots organization member there. But I also cycle to us and I know few people come from NTU today. So I'm by chance sometimes dropping by. Originally I'm from Russia, from St. Petersburg. There are only two seasons in St. Petersburg. If you ever plan to go to Russia, it also applies. There is July and winter. So go in July. This is just one question that everyone asked me. I wish to go to Russia. Go in July. Know that any other time. And I'm very, very, very big fan of Kaya Tos. And like as a Singaporean in heart, I really enjoy the cuisine here. I'm currently in the Partner Solutions Architecture role in Databricks and I cover the entire region here. My degree is in engineering. So master degree in radio, links, television. I also work with TurboCodes. Those are like the parallel branch of deep learning networks that are dedicated to signal to noise ratio reductions. And I previously worked for Standard Shadow Bank in cloud and demo environment. And before that I was with EMC Icelon. This is how I got to meet Spark for the first time working with Structure Data and Hadoop. And before that with a few other vendors. Now I wanted to set the stage for this talk, showing you this slide which is coming from the paper Google published in 2015. It's called the Hidden Technical Depth in Machine Learning Systems. You see the greenish dot in the middle which says the ML code. This is actually the proportion of time Google's engineers in 2015, I think still till today, spend on developing the machine learning code, the deep learning code versus other things. What are the other things? What are the largest boxes? You can take a look at it. You see that it's data collection, serving infrastructure monitoring, process management tooling. So the actual black box of the word, the artificial intelligence deep learning ML, that's a fraction of time. Most of the time Google themselves spends preparing the data, mumbling the data, merging the data. So it's the reality of today. So the current state of things is we live in a very siloed world and you've seen it in the previous speaker's presentation as well. So there were multiple systems involved in data preparation. They worked with, I believe they had a lot of Google Cloud Platform technologies, the previous speaker shared, they had predominantly GCP based solution, but you may pick up any of the databases, any of the store systems as an example, and they're not directly connected to the deep learning libraries, deep learning libraries that are able to create the valuable outputs for the business are not directly connecting to those sources. So there are different types of systems, there are silos in between them. And the good example is, for example, TensorFlow that only recently started to connect to the external sources. We'll speak about that in a minute. But also, the technology impacts the way that we work. And by having those silos, we then organize different teams. When we organize different teams, the teams have to start communicate to one another in some standard manner, so they create procedures. Procedures gradually slow down businesses. And with more and more procedures, with more and more procedures, businesses have to go for more and more approvals, reviews and exceptions. So the extraction of the value from the data becomes slower and slower and slower. The handles here on this slide between the teams is usually either in Jira's. And remember, we've talked about, you've seen this slide that was presented by William, I really liked it. There was this Twitter, a Twitter quote that said that it only takes like several hours to create the model and take several months to actually deploy it, because you need to talk to people, create Jira's, approve it, sign off, and then do the pull requests and actually commit it to the to the masters. Also, by having more and more silos, you're losing the ability to analyze the changes to track the differences. There were a lot of questions today about like, how do you see what changes to the model compared to what changes were in the data? How do you track all that? Well, you have multiple systems, multiple silos, multiple teams, you completely lose this visibility and definitely different infrastructure. With only, with only five systems I've counted in the previous speakers presentation, you still can kind of live, but in production, for example, on premise environments or heterogeneous environments, you may end up with like 15 or 20 of them. So the data though keeps on emerging and there's a variability of data that grows in different kinds and velocities and types. And the current idea behind Project Hydrogen, which is the Apache Spark version 2.4, is to start to unify these two worlds together to start breaking down the silos in between big data processing and machine learning. So with all the connectors that Spark has, and Spark has 50 plus quite stable and fast connectors to different storage types, including streaming batch. And even if you can connect to Elasticsearch, someone asked me today earlier in the day, like, can we use Elasticsearch? Yes, you actually can connect Spark to Elastic and query Elastic as you wish to. With all the capabilities that Spark already has, there are just few things that needed to be solved that would unify the portion about the total platform that would be capable to do the distributed learning, AI, and the data prep. So there's actually two thoughts that make sure I'm trying to skip a little bit. I think this is the history of how a lot of projects that we're all acquainted with emerged and developed over the last several years. So you can see on the left hand side, the Spark community started with the where Spark started was the MapReduce in memory, just a speeding up MapReduce, exploiting usage of the disk and being able to process large sets of data in a distributed fashion really quickly. And it started with the RDD API. It was like a very, very new API. No one was really figuring out how to quickly write a new code for it. And then it started to adopt more libraries and APIs to simplify the usage of Spark. So first, the APIs in Python, in Java, in R, started to appear. Then there was the new concept called the DataFrame that appeared. And it made it very easy. It was, it was in the unification single common denominator. They did the unifying pit that a lot of the lot of different skills from different types of background working in one team to use the same execution platform. So the person with Python background, or with Java background, or with our background, are now able to use Spark and teach one another, share the skills. So that gradually went into the Spark, in Spark's history, in Spark's world, that gradually went into the production shipping code that was ready and the one that you probably use right now. It then was extended by structured streaming. So one other interface that was available in Spark from very early days was the SQL interface. So you can actually query Spark just as if you would query the regular SQL, NC SQL standard database. You can query it using SQL. And then on top of SQL, the structured streaming API appeared, which was the same approach, but to the micro batches. Instead of querying the huge data sets, you split them into small, tiny pieces. And then within each, you can look at them from the perspective of like a second batch interval, micro batching interval. So this is how you work with the streaming data in Spark. At the same time in the AI and ML community, they were the same development. So they were first foundational projects like Pandas and NumPy. And on top of that, there were some performance optimization projects that appeared like XGBoost, GLMNet. Then there was a fundamental big development by the team behind scikit-learn. And slowly gradually we appeared to the TensorFlow's beginnings. Then TensorFlow was a little bit too low level for many of the data scientists. So then Keras started to simplify the way that the TensorFlow models are run. So it allowed to create the layers in the neural networks in a much easier, much simpler way, although below the scenes using TensorFlow. So you can see that it's all about converging. And definitely then there is a side project in TensorFlow that allows to access the data right now. It's called tm.data, tm.transform. So it all kind of converges into simplification and the ease of use. And what's next, the big double, the two question mark is, we don't know, time is short. But it looks like it all converges into something that would be the best of the two worlds together. And the communities start to speak in between one, in between each other. That's something we're observing. The reason why these communities need to speak is because, as you know, Spark is very popular for big data processing for ETL, for distributed, very distributed with speaking about like hundreds of thousand cores participating in like, in some jobs that one of the large customers of Databricks run. We speak about processing large amount of data, which is very beneficial for DL, for distributed learning. So this picture is actually coming from a research paper presentation from Stanford. And you can see that the efforts you put into feeding your neural net with more and more data, they actually pay off with the performance of the model. So the more you can allow, the more you can allow to train, the more data you can allow to pass to the model pays off overall in the model's performance and the model's accuracy. So there are a lot of projects that we're trying to blend the two. They are not entirely unique to Apache Spark community. So projects, for example, Intel had big DL that tried to do the distributed deep learning using Intel CPUs and leveraged them better. There were several projects from, for example, Yahoo. There were Cafe on Spark, TensorFlow on Spark, and you may actually find a lot of semi-orphan packages on sparkpackages.org that are trying to do the same things. For example, run Spark with TensorFlow in a distributed manner, but then people basically gave up after like first several commits. So definitely take a look at all of this project if you have interest in how to develop and where to stop. But what really we're interested in is two use cases to be solved, two user stories, and they come from a data scientist perspective. So a data scientist wants to build a pipeline that takes training events from a production data warehouse where the downstream kind of data lake ends up and train the model in parallel and also apply this model to already trained model to extract value from the same production data. So basically start inferring it in parallel into the distributed event stream that is already working there. So you've seen in previous presentation of the previous speaker, there's a whole deployment on top of Kubernetes that they have to work with by starting in a completely different source of the data, working with different siloed systems, it all ends up in Kubernetes in the end. This is the example of how segregated different infrastructure layers in that environment are. And the user stories, the user wants simplification, so the user stories to have one system that allows to do both of these things. So when we speak about the distributed training, this is the reinforcement of the same user story. So we want to load the data from multiple different data types, and Apache Spark is one of the best solutions for this user story. And then at the same time we want to fit the model, train the model, and do it in distributed way, and there are multiple frameworks using GPUs right now, for example Horovod for distributed TensorFlow. And they are then giving us the model that we want to productionize. And when we have a productionized model, what we want to do is we want to have a live stream of data that we connect the model to. But as I mentioned, there is a disconnect. For example, Horovod does not allow you to read directly through Kafka. So you have to create some sort of interim sync, some sort of interim load step, and also orchestrate this connectivity with some scripting, some additional engineering solutions, some CICP pipelines that are not essential for your extraction of the features, for example. So the afterthought for that is the glue code that we also seen in the previous presentation. So there's quite a lot of orchestrating, non-valuable projects that emerge, and they are sometimes maintained by engineering in larger organizations. They are maintained by the same data engineering team. Sometimes they are outsourced, but that connection is right now not realized natively in one platform. The reason why we were not able to really realize the same, realize both the working with the big data and the distributed training on Spark before version 2.4 is just the way the tasks are scheduled originally in Spark native scheduler. Spark is known for its so-called embarrassingly parallel task scheduling. So when we have a big data set, we partition it into multiple partitions, and then we'll let each individual compute unit of the Spark cluster work on its own portion. And if that particular worker, they're called workers, if that particular worker fails due to some reason, the task is getting restarted and it does not impact the other tasks. It's the idea behind embarrassingly parallel scheduling on top of Apache Spark. Now what is needed in distributed training is a little bit different. So to illustrate the task failure in our first distributed system, which was the embarrassingly parallel scheduling, we have a green task that crashed and it did not impact anything because it was designed from day one to be tolerant to system partitioning. In distributed system, you actually have to restart all the tasks together because they significantly rely on one another to communicate the proceeds, to shuffle the data between them, to update them on the stages of their job. So you cannot really restart only one task in a distributed learning, distributed TensorFlow system, for example. You need to restart all of them and that implies that we need a different type of scheduling. So the solution to this is something that was introduced in version 2.4 in Apache Spark. It happened several months ago and it's something called barrier execution mode. So if you go into the Spark's home page, you may find all the release notes always on the Apache Spark project and then it's the top feature about this release. So it says that this release adds a barrier execution mode for better integration with deep learning frameworks and introduces some additional higher end functions inside this mode. But not only that, there's also some improvement in Spark 2.4 that allows to use the defined functions in a much faster way than something that is labeled under the epic called optimized data exchange. And last but not least, there's a new way to... it's only right now available in Spark's own standalone scheduler. There's a new way to communicate the capabilities of workers, of the compute elements in Spark cluster, so that your task that you schedule on top of Spark cluster is aware about the underlying hardware. So it's aware, for example, that there is a GPU of a certain type or so on so forth. So these are incremental improvements in 2.4 in all group of them is sometimes referred to as the project hydrogen. So this is the idea behind it. I will demonstrate a little bit more about the barrier execution mode. But the idea is in the scheduler, in the mathematical, in the theory of scheduling, there is something called the gang execution. So the gang scheduling, gang execution is like either one whole gang of people, like adversaries, they go and do something really criminal or no one does. That's the idea behind the barrier execution mode. The realization, you can take a deeper read about the epic. There's this big JIRA that went into the release note. The idea behind it is to allow to start all the tasks in Spark from some point, only from this simultaneous perspective. And we'll wait for all of these tasks to complete and then get back into the embarrassingly parallel. They still operate in the regular Spark mentality, but from some point onwards you just say, from here start all tasks together and look at the way that this set of procedures is executed and once they're all completed come back to the regular scheduling. And also it allows to establish a context for the other jobs to be aware what's actually scheduled inside this barrier. And I think it will be a little bit clearer when we go into the demo, but the API to barrier mode is so far realized in Scala and Python. So there is nothing like that in SQL. If you want to call out for the user defined function or something like that across the dataset, you're still bound to Scala and Python for now. You have inside the context for a particular RDD, inside the barrier for a particular RDD, you have some additional functions that you can call. For example, you can call for the IP addresses of the workers that are currently scheduling, they're currently executing the tasks in parallel. So something like creating a cluster within a cluster that only does a distributed game execution job. So I'll go forward and demo it in a little bit later. Let's skip this one. Yes, you were able, for example, here is, it's important, for example, an API in an MPI. MPI is the way that in in TensorFlow, for example, you start the distributed learning. You have in TensorFlow, you have one single master host and then you run, you designate it. It could be not aligned to the regular master host, the so-called driver in Spark. It could be actually quite different and you need to be able to execute the TensorFlow job. You need to be aware about the, for example, IP addresses of the hosts participating in the distributed TensorFlow job. So you are able to pull and get the information about that sort of like cluster inside the cluster. You're able to get that information through the context within the barrier and then launch the usual MPI, launch the TensorFlow in the usual MPI way. So far, as I mentioned, it's only in the standalone, but there are already, you may see that there are several tickets, several initiatives to talk to the other schedulers where Spark is executed. For example, to Yarn, to Kubernetes, to Mesus, to also allow this information to get cascaded back into the Spark environment to create the parity in the project hydrogen. Speaking about optimized data exchange, optimized data exchange is when you have a particular Spark type data and usually it's represented by the DataFrame API. You read with all those 50 connectors, you read the data into the DataFrame API. It's very convenient to work with it. There are very simple functions to change the data, to expand it, to enrich it, to extract features out of it, but most of the deep learning frameworks, they operate in different data types. Usually they are native to Python. They are not aware about the DataFrame API format. One of the things that could be done definitely is just to write it out, wait for the right operation to complete, and then read it back, which is a waste of time, waste of energy because Spark operates very fast and it's optimized to be working completely in memory without touching the disk. So that's not a good idea. The usual approach is to call the transformation to the particular set of data, like for example for one row out of a huge one, and then pass it over to Python in the call using the user defined function. So the calls to TensorFlow, for example, in Spark, they are also done against the data invoking the data using the user defined functions. And the problem with user defined functions, they were introduced quite a few years ago, I think it was two years ago, Spark 2.2. The problem with that is the latency of taking a single row of a huge data set and converting it and then working on it in Python world and then returning it into Spark and then, for example, writing it out with all the possibilities that Spark has or acting at it at this data as if it's a streaming data. And when we speak about latency, the measurements that were done is about 92%, 90% to 98%, this was the estimation, is the exchange of the data formats versus the actual execution of the Python code. This is just the difference in between the the data types inside Spark natively and the world of Python where most of the distributed learning libraries operate. So the project hydrogen's initiative about the optimization to this problem is to introduce the vectorized data exchange and the idea behind it is to send the column. So it's similar to column in a database or column in an exchange, so the user defined function is executed against a set of columns, so why don't we send the entire column out of the data set that already has been prepared, featureized and stays in memory of Apache Spark cluster. Why don't we send the entire column of this data into Python UDF and then wait for it to return and then merge it backwards. So the kind of behind the scene realization of this initiative was done using the arrow. The community behind Apache arrow actually has a lot of different transformations already and they had established like the framework for different other Apache projects to start using it, so Spark project just partnered with them and leveraged the transformations that arrow has behind the scenes to speed things up. So when you call the two pandas rv or two arrow rdd as the user defined function, Spark handles it transparently for you in that columnar manner so you're not wasting and whenever I say spark, I mean spark 2.4 project hydrogen handles it for you in the transparent manner, it happens all behind the scenes and the re-usage of the already established protocols by arrow community and also some additional engineering that was done by Apache Spark committers allowed to improve the calls to use defined functions like 200 fold 300 fold so you can see the comparison between a row at a time call to the user defined function versus the call to the pandas udf with the vectorized columnar vectorized columnar format and yeah different lambda functions were used like lambda plus one or do subtract mean for example from a single column. So I want to switch to a very quick demonstration of the barrier mode. I think it's one of the very most very important aspects that is there in 2.4 so I have a couple of notebooks here and I want to I want to show you guys also the is this UI familiar for you you guys can you raise your hands who've seen the Databricks UI before because I know it's an Apache Spark made up so can you raise your hand if you've seen okay Databricks UI who's seen it yeah quite a number of people that's pretty cool thank you so the idea behind it for those that didn't raise hand is we at Databricks we are shipping the unified analytics platform which only works on the cloud so you cannot install Databricks on premise but the idea behind it is it's a collaborative notebook space that is a very very powerful way for people to work the code together and behind the scenes there is its its own cloud native cloud optimized scheduler and an Apache Spark core fundamental with additional optimizations to work with connectors with native Azure Microsoft Azure storage types with native AWS storage types and additional optimizations from the performance so that's very like 10 seconds explanation of what is Databricks for those that are not aware so you see the notebooks in here I wanted to when you go into the notebooks you have to attach them as usually in Zeppelin Zeppelin notebooks can you raise your hands yeah okay some of you use it so you attach the notebooks to the cluster so you can execute some jobs you you connect the code to the execution on top of a particular cluster by by by attaching them so I have a cluster that has several at this point this is a a native us environment and it has four worker nodes each one has one GPU and 60 gigabytes of memory the notebooks attached to it let's go into this first one distributed training are executing some spark code to do distributed learning using Horovod this is the regular MNIST example anyone remembers the MNIST the digits recognition samples okay sorry it's the first time that I'm at this meetup so I'm trying to gauge the the witness awareness that's good so the idea behind this is we have a set of numbers and they're already being captured as pixels like in a in a picture you have every black and white dot and you can create a long long list of rows in a very large database to represent the graphical data set on top and then you can pass it to Horovod to train it against the label set to train model and to know to um to then output the predictions so we have a table already the table has if we go into the uh I need to show you I guess the the table with the digits so you so you trust me you don't just take my word for granted so if we go into the low world no Horovod hydrogen MNIST this is what every individual digit looks like so these zeros are like black dots and then there are like the other the other grades of grade so if we start the job and train the Horovod model using all the four nodes let me just run them all what happens here we repartition we read in the we read in the entire table into the spark data frame so that data frame is then repartitioned into two partitions and in spark the amount of rows in you have a huge data set you partition it into smaller chunks and then this is the actual scope of work for every individual spark executor the work of the compute unit so right now I expect that there will be two executors actually working on the data set I write it out into an intermediate format here and then I execute the function against that data set that works on all works on two hosts so let's see we need to take a look at this stage okay so we we we have a jobs job stage one which is just kicked off and just waiting for the second stage it might take a little longer but actually it wouldn't care if I run at the same time what I want to do is like I want to use the same cluster for something in parallel and I'm not using any barrier execution mode here I'm just trying to infer the model that I already have and it's hidden in one of one of these functions predict using GPU I want to infer a model against the live stream of data that's in this case it's simulated against the column that will be the features and I want to create another I want to create another column in the same data frame calling it a prediction to what actually this digit means the hand recognize the digit and then I will call an action to display so what I'm ideally looking at is a two column two column table that has the label originally seen and then based on the columns in the feature based on the column of the features the prediction to what the real-time machine learning algorithm would would apply to it so if I kick it off by running it all and here like in these utils are all of these smaller functions actually they are all hidden so I'm expecting the stream to actually fail so let's see and the reason for that is that I'm trying to schedule four there are four nodes in this cluster four GPUs and I'm trying to schedule yeah it took a little bit longer than I anticipated this is complete yeah I've I've I've ran out of time I need to I guess like I took a little bit longer the let me actually stop this job and just rerun it for you for a second so that happens when you're trying to do the live demo sometimes and that's a that's a real cluster behind it I think what happened is the previous job commenced much earlier than I anticipated in the GPU cycle so let me what I need to do is I'll clear it all and restart it in a second so don't don't take my word for for it the idea behind it is the job as I anticipated has to fail here because it will try to use all the GPUs available in this cluster and there are four of them against the same against the same to to to run this test and two of them would actually be already occupied running the previous node books so let me clear it all up there's this function very very helpful function that is called the kill all API just in case there is something yet running on the cluster yep so let's kick this job off once more doing the prep work reading in the table with the digits um repartitioning them writing it into the party format and then reading the same um reading this party database uh with the model oh yeah there we go so I think yeah what what happened is just I was a little bit too quick for that so you can see the usual startup sequence of the hands-of-flow here um it does have a lot of details inside we will not go into it but basically right now I'm trying to I'm training the model on the four node cluster and if we look at the uh event timeline here I'm zooming so these are the two on the back of it these are the two workers that were selected and run the two partitions uh repartition two they run it against the GPUs so if we go back into the model inference without the barrier mode execution and we just say okay run um on all available nodes without any awareness about the barrier mode um run this code and infer the model that already exists against the live stream of data so what it would try to do it would try to infer also on those nodes that are already already got the GPU occupied because it tries to schedule four tasks and at this point it should fail I wanted to fail yay this is the kind of like the opposite example of the usual demo expectations you really don't want to things for for things to fail so it fails with the yeah anticipated GPU occupied error and the idea is we're like we're actually not able to complete the the task that Spark wants because underneath the below the hood there's already GPU busy so um let's stop it all uh let's actually go back to this job clear it all and clear all the possibly running MPI jobs by killing them all so if we repeat the same experiment with the barrier mode now the only thing that was added is this so it says also repetition the same data set into two but create a barrier for these tasks and distribute the information that you are using these two hosts for the distributed learning deep for distributed deep learning scenario distribute the information to anyone who is interested to schedule any jobs in parallel so it's a cluster inside the cluster so let's run let's run this let's take a look at the execution so you can see that because there are two partitions to the training set so yeah right now there is a yeah two tasks going on and if I'm running the model inference that is aware about the barrier context um I actually don't do anything so Spark does it for me so once I created the barrier for the two tasks in my previous parallel job the other jobs that I scheduled on this cluster they will be aware which nodes are currently occupying the GPUs so when I'm using the different notebooks in this case I'm still doing the same I'm reading the stream and for the every micro badge that I read I'm inferring it using the udf so now I don't want it to fail so yeah you can see the number two and number eight in this notebook and number nine and number ten are different executors that were picked up so we scheduled although we had the in Spark mind we had all the nodes available for scheduling we only scheduled two because we were aware that there is a barrier concurrently doing something on the GPU enabled hosts so that's the idea of the barrier mode and it just appeared like it's like two months they released 2.4 in in november the first talks about and I'm not the author of the presentation the first talks were actually by the committers like late summer they were talking about that so just went to to introduce this concept of the barrier execution or two this is the beginning of a very big progress because you're now able to mix and match the power of spark with the nature of their distributed deep learning and the last bit is the accelerated aware scheduling it's still in discussions you can see it's marked as the pending vote here the idea behind it is you can only specify those particular type of gpu or some technical some some capabilities of the host for further actions that you execute in spark cluster and it will execute these tasks only on those hosts that conform to this criteria but it's again like it's scheduling it's actually very dependent on the scheduler so there is the work going on to talk to the messes community to kubernetes community and to yarn community to actually make it happen to make this awareness presented back into spark and the timeline right now we have the barrier execution mode and it's available straight out you can also play around with it on Databricks platform the next big steps will be to integrate more and more functions into the barrier execution mode to make it more easy for example to pull out the additional context information and then going forward in spark 3.0 there will be optimized data exchange and the completion of the accelerator aware scheduling so that's the nature of the project hydrogen it shows you why we actually are rebranding this just Apache spark meetup to a spark an AI meetup because it's now getting into the single unified platform that merges the best of the two worlds together i know i was running a little bit short of time and we actually are supposed to end five minutes ago but still if you if you guys have any questions please raise your hand i'll i'll be happy to address those