Good morning, everybody. I'm Ty Lee, the chair of this committee on designing a strategy to evaluate the fifth national climate assessment. I'm delighted to see the people who are here in person in the room and and glad to see the people on the screen. We're joining us virtually. We have a busy two days that we're starting and we'll have some will be I think we'll have, we hope to leave tomorrow afternoon with the report largely the architecture report largely set for us to move forward so that's an ambitious goal that that's going to be we're going to have to work collaboratively with each other on as well. We will, you know, we're gathered here to think about to gather information first from the USG CRP from the previous evaluations that have been done and to touch base with a couple of the stakeholder organizations that will help us to get a perspective on perhaps the most difficult question who is the audience of the NCAA. We're going to try to craft a report structure, organize ourselves into subgroups to write the different components of that report. We're going to try to identify issues that are in front of us. And I think, boldly, tomorrow we want to come up with some preliminary conclusions and recommendations. So, get ready to be to draw conclusions, or maybe jump to conclusions is the right way to say this. Again, I wanted to call on Steven Stichter from the board on atmospheric sciences and climate to explain a little bit of how this committee fits within the structure of what the National Academy is up to. Stephen, would you say a few words. Great. Thank you, Kai. Greetings, everyone. Steven Stichter with the board on atmospheric sciences and climate. It's telling me to join audio and just reminder to everybody when you're when it tells you to do that you can ignore it. And we appreciate your, your time and your effort that you're spending here and over this year to move this forward. So, atmospheric sciences and climate is responsible for a wide range of topics and disciplines related to atmospheric sciences from atmospheric chemistry through observations through weather, as well as climate. The collaborative work that needs to happen in in this committee and representing that we almost all of the climate work that we do is in collaboration with other parts of the National Academy so we're pleased to be able to do be partnering in this in this study with the Department of Environmental Change and Society in our division on behavioral and social sciences and education. Brad Cheney is with the, with both. We have two partners in Dbass. One is the board on environmental change and society and second, the committee on national statistics, which Brad is a member of the staff there. So when we, we appreciate what you're doing. I wanted to give a little bit of a grounding for some of the connections for this work. The board on atmospheric sciences and climate and becks have been supporting the US global change research program actually since before it existed. We provided the National Academy provided input on the development of the Global Change Research Act of 1990 and provided ad hoc support for US GCRP from its inception. And then since since the about a little over a decade ago, there's been a formal advisory committee to the to the National Academy to the US GCRP supported by Bask and becks. We, in that work we provide advice and guidance to US GCRP in his global change work and then their, their mandate is in the process of being expanded to also encompass services as well. We have also in conjunction with that work with US GCRP provide external reviews for the national climate assessment. Kristen was on her second volunteer commitment in a row. First having participated in the recent national external review of the national draft national climate as fifth national climate assessment. The project right here that you all are participating in is one that we have started we started talking with US GCRP in about 2018 about getting this going and some things just take time. And so we're really pleased that this has is underway and look forward to the results of your work informing not only a follow on assessment that is to be conducted. But also US GCRP is in the process right now of standing up a federal steering committee for the sixth national climate assessment. And so the kinds of guidance and insights that you're providing about how to think about users and uses and consider the uses of the national climate assessment. We'll be taken into account as they are planning for the sixth national climate assessment in addition to its implementation in a in an actual assessment of the fifth national climate assessment. So thank you again for your for your contributions you're volunteering for this activity and really excited to see what comes out of this meeting and also the guidance that you're providing to US GCRP. And going forward for the national climate assessment and all of their assessment work so thank you. Thank you Steven. And so members of the committee let me just remind you that once we have our guests in the room. I ask you to go around and introduce yourselves very quickly. Name affiliation, perhaps what area of interest you have with respect to the question of evaluation and the national and the NCA five, but that that will come. That will come a little bit later in the morning. I turn now to first to introduce Brad Cheney from the National Committee on statistics as Steven mentioned. Brad is here as a staff person who has considerable experience on evaluation. And he's certainly been very helpful to me in thinking about how to frame the work of this committee. He'll be with us all through that he signed up for the duration so we're going to have him here. You've all met Hugh Walpole. And, and also Lindsay Mueller was in the corner. Hugh I wanted to ask you to say a few words about. Basically, the process going forward. When will be, when do you expect will be done? What are we going to do between here and there? Absolutely, thanks guys. So there's a couple of touch points I think that are worth us thinking about for this meeting today just because we have a great deal to do and not an enormous amount of time to do it in. So it's worth thinking about our short term goals in terms of assembling the report. So one of the things that this meeting was designed to help us support was putting together and finalizing the outline that we have for the report to give us a roadmap to go forward, the structure to build around. And so one of the one of our tasks is to to go through and look through that document that we've put together in the previous few weeks. And there's been a lot of really good conversation on that over the last few weeks as well, but I want to make sure that we have an opportunity to finalize that. And then in support of that will also be dividing in during this meeting into writing subgroups and so different groups that are responsible for different parts of the reports and once we finalize a report structure, we'll be able to split into those groups and discuss some of the key issues that are kind of dominating some of those different parts of the report structure. And all of this is in service of some kind of short term goals in terms of our timeline. And so our plan is to ideally in a meeting in a virtual meeting that we'll have in late March to have a sort of finalized draft that we can put together for the evaluation design. It's a relatively short period of time that we have about a month to gel on some of these issues and use that to support putting together a strategy that we can be comfortable with now it doesn't have to be perfect. But it does have to be in a structure that we're comfortable with as we move forward to making some of those recommendations and this again is all in support of hopefully by the middle of April, having a full draft that we'll be able to work on editing and getting ready for a process of going into our external review process in May of this year. And so, again, there's a lot to do in a relatively short space of time but it's a really, really good group. We've had so much engagement from all of you already so I am fully confident that after we come out of this meeting will be kind of being tall with a set of marching orders in terms of how we're going to put this together and so I'm fully confident that we'll be able to put together a really great report coming out of this. And Kai, please let me know if there's anything else you'd like me to cover in the meantime happy to do that. Well, that's, I think that's a good. That's a great sketch. What we have a question that we want to take up tomorrow. When we go through the outline of the report and see how we feel about it is question that, as Ariane put it, every undergraduate asks, how many pages are we trying to write here. And that's a question I don't want to raise. I don't want to, I don't want to explore that yet, because we don't, we haven't had a chance to talk to each other about the full outline. That's just a coming attractions. Right. Any other. Let me just pause here to see if there are questions that people have about what's been said so far. And so that we can, before we start focusing on the guests will have later this morning and early afternoon. Any questions. Stephen what while we still have you I don't know how long you'll be able to stay with us. A question that that came to mind as you were talking about what Basque does in collaboration with the other committees and boards. What perspective you might share with us and having worked with the committee to advise the US global change program for some time. What perspective you might share with us about what we're what we should be doing in this report. I say that because reading the summary of Heather Danzker's earlier evaluation, what struck me is that the durability of the issues with respect to evaluation that have been raised. The same questions we've talked about amongst ourselves and our, you know, online meetings are there in that in that evaluation of the third NCA that's three NCA is ago. And so I wonder what what help you might provide to us about why this what set of questions that that are in this our statement of task, how they came to be and and what you think USG CRP is ready to hear and to act upon. So two parts of that how they came to be those all of our work is as we're scoping out our work we we do that collaboratively with with the sponsors. So these do reflect both insights from Basque and the on the physical sciences and I don't want to draw draw to to firm of a distinction between the physical and social sciences because it's really built into what we we do throughout Basque that the social science how we actually move the information and knowledge that we have into practice is that is baked into what we are doing but so it was a collaborative discussion between back back and becks and USG CRP. So it does reflect the kind of information that USG CRP over a course of years so as I said the first time that we proposed, moving forward with this was around 15 and just through funding more getting them funding in place took a while and but those so these that those questions came from collaborative discussions between the National Academies and USG CRP. I think that there are one of the things to know about USG CRP and Glenn is this deeply from having been in the midst of it is USG CRP is a coordinating body. So the actual work. There are some things that they actually implement such as the National Climate but much of the work of USG CRP is really trying to get as they call it coalitions of the willing together of finding agencies and groups of agencies who are willing to pool their resources and their, and their attention to specific issues. So, one of the one of the challenges that they have and it's a, I think it's a positive in the end when when the work is is completed is that they have brought together agencies to do to move forward on their goals. I think that one of the things that we may hear from Allison is simply that the National Climate Assessment is a beast to move forward and as over time it has grown in size and the number of participants and authors. And so I think one of the even the challenges, the desire that they have for for moving forward on some of these questions around how, how it is implemented how people are engaged. I think we some of the engagement questions if you look at the fifth National Climate Assessment and the number of authors and the range of authors. I mean, the move in that direction and yet I think there are just some practical and logistical challenges that they're, they're running up against, but I think that the work that they're doing for instance, now of having just released the fifth and national climate assessment and they are already moving forward on the federal steering committee for the sixth is first meeting in March, and they are looking to have a director for the National Climate Assessment on board this spring, which is faster than they've been able to do it so I think they are set up to hear some of these questions and have some extra time to hopefully move forward with them. I think Alison Krimmins is a phenomenal resource, and you have an opportunity coming up very, very soon to speak directly with her and I, she's not afraid of hard questions so I encourage you to use that opportunity to hear directly from us on the CRP. Questions for Steven on this. Because I think an undercurrent Steven of our conversations within the committee is a sense of that we want to question this. The easy answer is because climate affects everybody, the National Climate Assessment should address everybody. And the, the secular trend over the last five assessments since the 1990 has been in fact expand the audiences that people are trying to reach out to. I think there's some skepticism in the committee in this committee about about that. That's of course up to the agencies in the US GCR P to decide, but raising those questions is something that I think we are inclined to do so I just And that's something that I think we need to talk about within the, you know, the committee, the committee's mind is by no means clear on this I just wanted to identify that as a theme, and ask you Steven if about about your about your perception you watch this more closely over a longer period of time than any of us have done, except for Glen as I suppose who's been who's been watching this all along. And it's that that I think we're going to wrestle with in the next couple of days. Yeah, and a couple of comments on that. First, I know having had Allison in a, in a different meeting yesterday related to the US, the advisory committee, she is aware that some of these questions are coming to her today and so I think she's poised to have a really robust conversation about the users of the national climate assessment. So that will be great to hear that directly from her. I think you also are coming from a very unique or you are a specific perspective, which is evaluation and bringing that bringing those those questions about the users and uses and that the challenges or the challenges that are posed by not having a clear framing for those users. I think that's very much falling within what you were asked to do is about how do we do an assessment of those users and uses so speaking to the the limitations or the challenges that presents by not having more clarity around that is feels very much within what you're asked to speak to. I, there are some clear audiences within the legislation for it is the, it is the document that federal agencies use and reference when they are bolstering and referencing work on climate. There's a clear audience there but I think the question of how to make this massive, massive effort and really important document meet a broader audience I understand that that poll. But that's a that's a tension that I think you're, you have a good opportunity here to talk specifically with USG CRP and also to point out the downsides of that of being too broad or challenges with that specifically within the context of trying to evaluate whether this document meets those the needs of those users. Good, thank you that's, that's, I think that's helpful. And for those on the committee who have not been involved in the work of the academies before. We also have multiple audiences to be thinking about. Addressing the USG CRP, which formative the station set set a statement of task behind them that as Steven said they are coordinating body. We're also addressing the agencies. And the agency representatives who work on the NCA. And it's not unknown for Academy reports to get the attention of the Congress and stakeholders on questions like that we might raise about the levels of resources that should be devoted to the evaluation. So we, as we as we get into thinking about the outline of the report in the next couple of days. We want to think about multiple audiences as well that that we are addressing. And, and how we want to, we just want to be aware of that I think as we as we go into it. Steven. One more comment just about the multiple audiences. I would say. I would also note that within USG CRP, you also have multiple audiences. You're asked to talk to the national climate assessment. And yet USG CRP will themselves be very clear in saying they have not generally done a very good job of or they have not been consistent about building monitoring and evaluation into their programs broadly. We had a conversation at the advisory committee in the fall around climate services. And one of the questions that they asked the advisory committee to help them think through was as they are stepping into climate services, how do they think about building in monitoring and evaluation from the start. So your charge is specifically to the national climate assessment. But one of the multiple audiences that you who will be reading your report is folks within USG CRP broadly about how do they think better. And from the inception of activities of building monitoring and evaluation into that work. Great. Thank you. Dennis. The charge does mention the national climate assessment and other USG CRP products. Great. Great. Thank you. I. You've read that more deeply than I have in recent times so you can use those. You can use those on trace to speak to those broader audiences directly. Great. Okay. Thanks for bringing our guests. Absolutely. If there are any further questions I'll just run and grab our interests and guests and then we'll begin letting our guests into the zoom as well. So, brief interim for just a moment while I go grab them. Welcome this is a. There are no assigned seats so you can just. Thank you. First where you want to. Good morning, Allison. Welcome. Hey, welcome. Heather and to Allison Cremans. Before we get started, I wanted to go around the room so that the two of you. As members of this committee and that we have a couple of members who are online as well, not here in person. Let me begin with them so that you see them. You see them at the outset since you're not going to see them in person. Michelle, let me start with you. Sure. Hi, hi everyone. Michelle. I am unmuted on my computer. We're still hearing you so. Okay. Let me ask a rouge. Let's try a rouge to see if she is. Hi everyone can hear me. Yeah, okay. I think we can hear you. Good. Okay, so maybe take it back to Michelle. See if she, she's still. Hear me now. Sorry. Oh, good. Okay. Something, something released. Apologies. Michelle Myro. I'm a senior information scientist at Rand. Nice to meet you all and apologies for not being there in person. Rouge. Hi everyone. Good morning. My name is Rouge and I am an assistant professor at Loyola University Chicago and really excited to be here as well. Great. Let me just start with Arianna and we'll just go spatially around there. Good morning. I am pins and then with the Army Corps of Engineers. Kristen. I'm Kristen Tim. I'm a research assistant professor at the University of Alaska Fairbanks. Scott Califatis. I'm the deputy university director of Northwest cast based at the University of Washington. Jessica Cronstadt at the planetary health alliance. Hello, Glennis low as the global change Institute. I'm Kathy Segers and I'm a professor in the economics department at the University of Connecticut. I'm Carlos Rodriguez Franco for a service research and development here in DC. Matt. Hi, I'm Matt. Hi, I'm Matthew Gribble. I'm the associate chief for research and occupational environmental and climate medicine at UC San Francisco. Stephen. Stephen Stichter with the board on atmospheric sciences and climate at the national academies. Part of the community of national statistics. I'm Hugh Walpole. I'm an associate program officer with board of atmospheric science and climate and I'm the study director for this project. So I'm just helping the committee coordinate. Hi, Ann Gallagher. I'm with several institutions and I'm an interdisciplinary person on this committee. So I cross all sorts of exciting boundaries with the park service and I'm also with Old Dominion University. And Lindsay Mueller is in the corner there, making sure that we can be seen and heard. Heather, welcome to you. Would you say a few words of introduction we've already read some of your previous work. So we're glad to have you here. Thank you so much. So I'm Heather Dansker. I am the founder and principal at Dansker Consulting sole proprietorship. It's consisting of me and my partners who I bring into various projects, largely focused on program evaluation and I'm based in Arlington, Virginia. And I'm so happy to be here. Thank you so much for the invitation. Let's see. Okay, yes. Okay. Actually, you can, you can keep it there, but I'll give you a little more background on who I am and what I do. So, um, I've been in business focusing on program evaluation working largely with federal partners and federally funded efforts. And I've been doing that for 11 years or so. So, part of that I was in management consulting, working with federal agencies for four years and part of that I was in a research extension, dual role at a land grant university Cornell, my alma mater, focused on both doing systematic reviews in a lot of cases and then also doing the communication and in that case it was around chemical risk and human health. So I've had a little bit of a dual career in the sense that I come from both the social sciences and the physical sciences. I worked in chemical risk and toxicology for over a decade. And, but I've also parlayed that with looking closely at the social and human dimensions. And eventually have transitioned away from chemical risk to focus in program evaluation and I'm, I'm enjoying it tremendously. Um, since I've gone independent, I have led teams or worked independently across a number of fields I originally, and for many years, even decades consider myself firmly in the environmental space, but of course that's that term is evolved, as we know with with the challenges associated with climate and how broadly underpinning those are to really everything so I work a lot in sustainable ag these days. I also work with us GS, I work with many parts of USDA but internally and as an external evaluator on on large multi institutional research initiatives at universities. I also did a stint, interestingly, for the defense security cooperation agency and deputy under secretary for policy DoD during the COVID era, where I led a small team. Well, we were the evaluation methodologists as part of a much larger interdisciplinary team with defense subject matter experts looking and conducting three different high level strategic evaluations that were mandated by Congress to DoD as part of the evidence act. So, there's a lot to say about knowledge management, of course, and I'm guessing that our discussion will will flow into that kind of arena as we go so enough about that. Next slide. Okay, so just an overview of what I'm going to do here. I'm going to talk about the guiding questions that are on our agenda to keep in mind, however, my presentation does not answer those directly, but rather it's just a reminder to all of us of what we're here to think about and consider, and I will be touching upon some of them. Others I think are out of the realm of what I can speak to today, but I'm going to present those so we can have them in our in our brains. And so my team, back in 2016 was commissioned to do a retrospective evaluation of the process, the development process, and the products of the third NCA as you probably know. And that was, I had multiple partners, new knowledge group, and others that were subcontractors to my team, total about 15 people over the course of about 11 months. I'm going to talk about that process of evaluating the process and its products. So I think what the questions that are of most interest to this committee, which are how did that go. What are the questions and how 10 years later really or eight years later do we think about evaluating different aspects of what the global change research program creates that that climate information that climate knowledge those products, including the NCA report itself. So it's hard, even for me, and I think for a lot of others to actually separate that from thinking about what the NCA is how it's made what it does. And of course they're related but I think we don't want to get to find the down the path of what we found that's, that's, that's many years in the past now, I can speak to some of it from my memory, but we won't be focusing on that. Instead I'll be highlighting some of the strengths and challenges and limitations of our approach, and then in and taking that into thinking about considerations for evaluating an NCA and, and I use that holistically, as the products of GCRP and its partners. Going forward. Next slide. Okay, so I rearranged these questions that are at the top of our agenda. I'm falling into several buckets and to help us kind of parse out what we're really thinking about here. We're thinking about the evaluation of the NCA. And again, either broadly or narrowly, either the report itself or that larger universe, the universe of product. And so we want to know about what some of those challenges are. I want to think about that broadly rather than narrowly. And we're interested in those types of stakeholders, partners involved in evaluation, which is separate, I think, from the question of who are the audiences for the report and I have seen in past and I'll get into this in more detail but I have seen in past outputs that sometimes I think that gets a little bit conflated, understandably, but I do want us to think about the users, which in the evaluation world is how we really think about people who interact with valuation products are users, and often primary users. And then so thinking about maybe and that's maybe a way of easily separating this from thinking about the audiences of the NCA report and other products. And then this broader question even of that larger climate information universe. There's this question of sources of climate information that users are using that audiences are using. And, and other gaps there and that's a broader question that I don't think I can answer I've not studied that I'm hoping some folks have perhaps at universities and otherwise, certainly an important question. Next slide. Okay, so a little bit about our process so we the as you many of you may know, there was a workshop back in 2014 around the time of the release of the third NCA. That brought together over 70 folks GCRP folks and others to think about how to go about evaluating the NCA. The NCA report the NCA process development process use were all considered in this. And the report has a nice. It really contains a lot and I think it's still very relevant. I read it again last night, and I think that it has a lot of good ideas in it. So I would not say it's dated however and I will I will be speaking of this further but I think there are their new frameworks for thinking that have emerged in the intervening years that we also need to consider and I think those can help us. And sort of bucket was in this report. Good idea certainly but I think part of it is, we'll see that a lot of what's in this report are the finer grain details of the types of questions we want to ask, and what we are interested in looking at. But I'm going to lay out a framework for thinking about that that that makes it a little easier to think about. Appendix G in this report had a list of all of the types and buckets, if you will, of types of questions that that this workshop yielded in terms of things to think about. It turned out to be a pretty suitable framework for our task which was to consider and evaluate the process of developing the NCA, and then also its products. And that's essentially what this workshop and, and this the framework that's presented in appendix G. That's really what that that allowed for. It did acknowledge that there are larger questions, and I want I'll get to that. So the arrows here in green are really the sections that we focused on to capture and assess in our work. I'll talk briefly about the structure of the development process, what it took to all the inputs and resources that went into it, what writing and review looked like, how communication group dynamics, what they looked like in this very large process that was very interdisciplinary and brought in more external stakeholders than previous NCA's. The benefits came out of it for the, the developers themselves and that's an important thing actually is there's a limited, arguably a limited set of scientific expertise in climate science we don't want to burn folks out in this regular process and so how what the benefits are there for that continued participation which is basically unpaid and takes a lot of work. And then the dissemination and access and, and the application outcomes of the report. That was really tough and I'm going to get to that because it involves getting the information from the users and how we did that and I'll speak to that. So next slide. So the methods we, we approached it from a mixed methods utilization focused approach which is really utilization focused evaluation is very commonly used approach which means that one of the primary premises of it is that you, you identify the evaluation team and it's the partners identify early on who are the primary users of this information what was this get what the purpose of this and at this time the purpose was to learn how it went to have some lessons learned for next time. And so those users were largely GCRP itself, the NC State Climate Institute for Climate Studies, the technical teams involved, and of course, any others of the external stakeholders and partners that were part of it at the other groups that may be interested in, and learning from that being part of that. Like I said it was focused on the process and the products. It involved two different surveys of both developers and users. The first survey was one of the challenges so we ended up using the GCRP's newsletter listserv. And that was approximately 5500 email addresses, we got very low response rate, very low. So, arguably, so low that you need to really question what the findings were from that and put those in context. We did get representation and we looked and calculated and analyzed quantitatively representation across these data collections because that was one of our charges. And we did get representation, however, it basically we, there's more to do on that front, because the user group was still limited, and we don't, we don't, we can't say that the users that responded to the survey are actually representative of the true user group of the NCI at that time. So, limitations there. That said, all of the data sets did inform what we were able to say about NCI use to some degree. Some of that came from, so alternative methods of looking at use were looking at doing citation and content analysis. I don't want to get too into methods because the methods really flow from those higher level questions of what your general purposes for evaluation investment. Who's going to be involved in that over what timeframe, then the methods follow from there. So methods are not the starting place. But since we're on the topic, I will say that when thinking about assessing use the time component matters because there's learning to be done in the moment of the development. There's learning to be done in different moments of use, but there's also learning to be done in the out years in an ideally optimal, the optimal landscape for when that product is being used before everybody moves on to the next one. Next slide. Okay. So, like I said, I, there's so much to say, but I think we're not going too far down what our results were. Instead, I'll just talk about some of the very highest level strengths, challenges, limitations and you'll see that inclusion, inclusion, inclusion. A strength was the inclusion of both the process of developing it and, in fact, that so many of those developers. And, and again, I would say weaker on the side of users but so many of the developers showed up to be able to inform our evaluation data collections. So the evaluation was in that sense inclusive. But the process itself was inclusive. The challenges, like I said, were that identifying actual users in a way that we could have confidence was representing full use of the product was very limited. Also, I would say that the essentially post hoc nature of conducting a kind of trying to capture a lot of things after the process and after the release of the report has definite limitations time matters a lot. And in fact, when we conducted ours in 2016 that was already several years even three to four years depending on where people put in most of their effort into into the development. And for for anyone that's a long period of time to really reflect on the nuances of the experience. Again, underscoring the idea that there is value in what is called formative assessment. Basically, side by side along in that process for for learning and process process improvement. Limitations again were inclusion. At the end of the day. We still felt like there were. It was not as inclusive as it could be. One of the main takeaways from that was that for disadvantaged groups and individuals perhaps representing those groups. Even as leaders of those groups in some places around the country, they do not have the resources and did not have the resources at that time to engage. As we know, of course, post pandemic, there are more ways to engage virtually which is helps tremendously. And yet, it's important to still think about whether they're adequate resources to be able to allow for inclusion of of disadvantaged groups. And again, like I said, doing and thinking about this as only a post hoc exercise is a huge missed opportunity for for that kind of learning that could be really optimized. If it is baked into the process. Next slide. So some considerations going forward. As I think probably most of you are aware, there's been culture shift for the better in this area in the area of thinking about assessing large federally funded initiatives and and who's included in those. There's, as of, I think, 2019 the evidence and policymaking act compels federal agencies, a set of federal agencies, I think those that fund with funding streams to really set up an evaluation enterprise. That includes evaluation strategies and frameworks and learning agendas. And I really want to underscore learning here. This is about learning. This is not an audit. People can certainly ask questions and people in groups can certainly always ask questions about what bang we're getting for our buck in terms of federal dollars spent, but, but it's also really important to underscore that all of this is just about learning and improving upon what the ultimate goal is, which leads me to the first bullet here, which is that embedding the evaluation into the end of the life cycle and thinking of that as a life cycle with these frameworks is really I think from what I understand and I've not studied this closely. But what I understand is still something that needs to be further developed. There's been there's a change and logic models that have been created as part of this endeavor that exists. There's, there's one in the workshop report. There are probably others. I would argue that those need to be looked at very closely because they can be very useful tools in clarifying the overall goals at any given time and it can be an evergreen tool. It can be fixed in stone. It will always be evergreen as our challenges around climate information and responding to, to the decision makers in this country around climate challenges evolve. The difference for those of you who may not be aware with the subtleties in evaluation tools, a theory of change is really a model. It can be narrative or graphically represented but it's a model that illustrates the relationships between how you think that an intervention is going to get you to a goal or a set of goals. And it is the logical relationships between those that are laid out. Whereas a logic model is something that is more finer grained to a specific initiative, and that can be in this instance for, for example, used with any given NCA report for perhaps, or other set of other constellation of subject that you want to look at with specificity in a given more bounded time frame. That is also something that can be evolved, but it allows for looking at specifics in a way that the theory of change illustrates essentially the logic of getting from here to there towards goals that need to be really worked out. And in this context, I think, with adequate open and stakeholder involvement. So, I think if I were to circle and read possibly one of the most important takeaways that I have come away from since I've been thinking about this for today is the importance of the development and prioritization of evaluation study questions. And really they tend to be sets of questions to optimize. I'm what I'm sure is a limited resources for evaluation investment. I would say, with all due respect that the initial evaluation workshop didn't quite get us there, it had a lot of great ideas. And, and it, and it has a lot of ideas for questions and aspects of the, of the entire enterprise to be looked at and inquired into. But without, and that can help get the juices flowing, but there needs to be a process and a framework for thinking. What is most like basically what are some of the big picture questions that matter here in terms of what these products are meant to serve and thinking about what buckets those fall into. And then having a real prioritization process for what's most important to fund here that will inform and improve upon this system. That I think might be a real top level recommendation here. And of course always there is the idea of like I said before, who's going to be using the evaluation product for what purpose. And that can help bound who's involved because I know the question about who the who who's involved at any given step seems very overwhelming and and again kind of conflated with the audiences of the report itself. But that's one way to really bound it it's like who needs the information for purpose. Next slide. I think one back. Oh, wait, hold on a sec. No, I'm wrong. Okay, slide eight. I need. Yes. Oh, I think I said, okay. So, excuse me. So, okay, like I said, form of evaluation, of course, some of the valuation is also important, which is really focused on outcomes. Sometimes there are questions in this space around how is this impacting climate decision making, and their arguments for and against looking doing outcome evaluation in most cases outcome evaluation is what matters. So, let's talk a little bit more about the process and the, and the activities. The, the, the, I guess, the arguments away from outcome evaluation are when you really do need to understand how a process is functioning, and the contrabutional nature of the outcome gets really lost in the bigger space and I think I've seen this this is actually I think the workshop report pointed this out that when you, if you are not trying to illustrate a causal effect. For instance, not trying to illustrate that the NCA report itself had direct impact on specific decisions in the US but rather trying to illustrate its contributions to the ecosystem of decision making that gets hard and some argue that gets diluted in their context of evaluation, it's very useful. I think both are necessary and important to try to do. But again, it depends on what your primary goal is for purpose. So, this is related to. And again, just one more note on the summative which is, there are different contexts for thinking about impact. There's the context of thinking about the NCA in relation to the larger climate information ecosystem. And there's also those decision contexts that could be regional, topical, very time sensitive responsive in nature, risk related questions and the science itself. And so this gets us into the knowledge management space. And there's these new ideas around that came out of the covert area actually in terms of data management, and thinking about how we take a sum total of knowledge at a point in time, and then adding incrementally to it for efficiency. And for rolling out of new information to audiences in ways that don't reinvent the wheel each time. And are essentially efficient and allow us to get up to date information when we need it. And of course, with AI and the systems that are starting to emerge, I think this may become more possible. Next slide. Okay, and then of course I can't not talk about inclusion, diversity, equity and these others important societal values. There are others I don't want to limit it to those three. These are, it's essential to also keep those up to date and to build those and into the fundamental nature of the framework for evaluation and the learning agendas and the goals that these processes seek to meet. So, the process of identifying and coordinating the stakeholders and partners are increasingly networked as we know that's one thing that is really emerged in these years and it's wonderful to see always more to do in that space always more to to grow and connect. But these, let's see, essentially that coordination will not go away it will always be there for different purpose so it'll be there for thinking about the evaluation process, as well as the sharing of the products themselves and ongoing learning overall. And I'm going to leave it there. Because I think we have things we can discuss before my time may have already run out. Thank you. Thank you. Okay, thank you. Thank you, Heather. Questions from the committee. And, and those who are online, please, I guess what I'll do is I'll, I'll ask Michelle and Rouge to raise their hands in the zoom. If you have. If you if you have questions but floors open. My question has to do with the slide with the seven buckets. You know so you've got the does everybody remember what that was. I just wanted to say before having read the assessment report. I didn't anticipate all those buckets. How did you come to reach those buckets, where you charged with all of the range of those buckets or were you also have advised like a group like ours. And those buckets came to be right so those buckets actually are very parallel to what came out of the 2014 evaluation workshop and the appendix it's an appendix G it's online. You'll see there's a lot of parallel. Those things are very parallel. We did have a process as part of evaluation with the. So, so we were. We were tasked it was our charge to evaluate the process and the products. There are other frameworks and other sets of study questions we could have had we could have gone a lot of different directions, but it was not our charge. We could have been charged with looking at the development of risk language for instance, which is, of course important and and how users interpreted the language that the developers. Developed right around risk and probability. That was not our charge. So our charge was actually looking at process and and products. Fortunately that workshop report provided a nice framework for that and the team that we were working with which is GCRP and the North Carolina technical team is to climate studies team. We worked with them together and decided, yeah, this is a nice constellation of questions that we are interested in looking at. We dropped a few, very few. If you do a crosswalk between appendix G and what we collected data on it's it's very few we basically included all those in both our qualitative and our quantitative data collections. Yeah, I had a question even in in reading what we read before this meeting, it felt like process was maybe like the larger chunk of what was evaluated and products were somewhat narrower chunk and so I just wanted to see whether you felt like that was a accurate perception and then I was also wondering how much you thought maybe the, the kind of constrained nature of the focus on products was a function of that period in time. And just the, the communication environment being a lot narrower at that point in time compared to what we see now. I agree that there was definitely more of an emphasis on the process, and that was sort of what was just agreed to by the group in terms of what the group was interested in at the time. I will say that there could have been other methodologies and approaches for investigating and characterizing use certainly I mean the. I did citation analysis, one can imagine sampling across that citation analysis for instance and then going deep qualitatively with questions around what the NCA meant to those users for as one example. Similarly, the GCRP newsletter listserv that was used for this, the user survey, how to range of stakeholders. But, but the challenge there is that just because they're on the newsletter listserv does not mean they were had used the NCA and in fact I think that was one main reason why we got a low response rate. Other reasons might also be that whatever email addresses were on that newsletter listserv may be institutional for instance and didn't go to individuals that might have had, have been compelled to respond to a survey for instance. Yeah, Kathy. Yeah, thanks so much for that summary it's really useful. And you mentioned that that appendix G has the framework that you used, but you also mentioned that things have changed since 2014. And we've learned, you know, new things and so I was wondering if you could just summarize what you see as the key things that have changed you mentioned a new emphasis or an emphasis on DEI for example, as one thing. So are there key things that you think have changed that we need to think about when we think about a framework. Yeah, I mean I think that if you think about, let's think for a moment about a hypothetical longitudinal system where we, we have a limited set of evaluation resources and we have to use them for purpose to answer key questions to make this all optimally functioning for the use of this climate information, right. So you might expect then to say and say you're allocated like we are with budgets to spend some of that over time in ways that are again optimized. Well, I would argue that it was, it made sense to learn from a very large complex development process. It was the most complex development process involving many more people over 350 people were involved in that process that was unlike any previous NCA development process. So to focus on the learning from process made sense at that time, it may not make sense then to keep asking about process, because some of these things we might say well it might be interesting to know how that's changed, but it's more important to ask different question sets around, for instance, again, like my risk example like the, the specific scientific and probabilistic information that these scientists are bringing to the language that gets that lands in the report matters to users. So I think that that closely could be a really important set of study questions that you could still come up with methods with qualitative and quantitative to, to, to inquire and, you know, and investigate. I do think that the emphasis on use, and, and how this is getting, but not just use but like how is it getting to people like connecting it to users is a big chunk of it, is it get connecting to users. So adequately I would probably just say no off the top of my head it's probably not yet. How can that be optimized. Then, okay great maybe that's the next step I don't know I don't I'm not going to recommend make that recommendation, but certainly optimizing use and getting into the hands of it decision maker decision making types of groups across this landscape. That could be the next thing in and of itself, and then looking at those decision contexts and asking important questions about what they actually need, and whether this is providing it. And of course there's the entire new constellation really I mean it's been evolving of the partners the networked partners who are using this resource as one of many resources that are trusted and foundational. And, and rather comprehensive in their work in those regional topical time responsive contexts. And so that all is part of thinking about what you prioritize for assessment. Grush. Yeah, first of all thank you so much for your presentation. My question is just a follow ups I think you said in the beginning of your presentation that the response rate was pretty low for the main sort of survey that I'm guessing you used to build your, or cultivate your findings. I was wondering and then you also said that representativeness was not an issue so even though the response rate was low. The data was representative. Could you just elaborate on that like what do you mean by representative like do you mean geographically do you mean under engaged communities. Right. Yeah, no, so there were two surveys. And the developer survey had a good response rate. I don't remember it off the top of my head, but it was, it was adequate. Plus we have the qualitative for the developers. It was really the user survey where we got low response rate. So the, the, the, the analysis of representation was done on on both. I believe but the on the development side. So across the 350 developers, we looked to see their, that was cross sectional across geographies topical areas, federal versus not federal, etc. So we got adequate representation. I believe we did look at representation of the user survey, and it did. Yeah, we did. I know we did because I know we had a chart on that. It did capture the range of stakeholder groups that were, I think, bucketed and like essentially there's NCA net, which was this stakeholder community that GCRP had fostered. And we did get more different types of stakeholders within that we looked and did a kind of crosswalk, I believe, between that community and who answered the user survey and we did get representation across those, but not necessarily given the low rate. That throws that all into question around who was actually using it. So, so yes and no is the answer. There's another, not a follow up, but a separate question. You also said that since the NCA 3, there's like more emphasis on learning and evidence. So I was wondering if you could also elaborate on how you're conceptualizing this evidence. And if you have any recommendations for the committee when we're thinking about what usability evidence looks like. Yeah, so, okay, so two parts and maybe in this, maybe a simple way of perhaps overly simplistic but way to think about it is thinking about the value that comes out of formative assessment versus summative. And that's formative. If, if you build an evaluation framework and evaluation approach that essentially walks side by side with the process and is embedded essentially in what it is evaluating. You have learning that is iterative and that comes out of that process. And that's formative and it can be used at in times, timely ways for improvement. And also just takeaways and learning like maybe those lessons learned can't be rolled back into improvement but are noted for next time. So that's formative learning. The evidence piece of that really can be thought of me can be thought of on both sides but let's just take summatively it. It needs to be considered when thinking about the overarching framework, getting from here to your goals and thinking about what your indicators of success towards achieving those goals look like that is an entire process. That I would argue that the NCA enterprise still needs to set up. It's really important because without it, you can't really prioritize what you're investing in for assessment. And so evidence then is about what you're focused on collecting and measuring for understanding and impact. And in the, if you think about it from the more techno credit if you will knowledge management data management space. It's ultimately about information systems and what you can't collect everything. What are you going to collect and track over time what's important to track over time and and and what's not. There will be cross cutting indicators that matter, but figuring out what those are is a process. I wanted to turn in a slightly different direction to focus on this use question because a lot of the language about use assumes that use is a kind of particular delivery of information to a particular decision maker. And the NCA is, as you mentioned before the USG CRP is a coordinating body. And so one of the primary audiences arguably the primary set of audiences are the agencies that that make up the US represented on the USG CRP. They do things with the NCA report and the information that and it's associated products in their own spheres. So the Army Corps of Engineers might take the work on a lot of resources and combine that with other aspects of the report of science in the NCA and reformulate that for different constituencies within the Army Corps. In that sense, use becomes a process, not an action. And, and I wondered, to what extent, your evaluation focused on that, because there's the, you know, you can imagine that working for the agencies in the sense that that you can retrieve some data. One more complicated and open ended question is what do stakeholders were not in the federal family. What do they do, how do they react to the NCA, which is also a process but much more diffuse, hard to get, you know, and hard to grasp one. I just wondered whether in thinking about use as a process in influenced what you were finding. You have to really open so we certainly did not narrow it down to just decision making. And so, in a sense, it is an input, it can be considered an input into other processes, like you said. So it was used across sectors. And the results that we did get were about. I mean, I have some of some notes here jot it down, but essentially learning and awareness building around issues in places and around topics. Certainly informing different types of planning governmental planning. There is an entire sphere of in which is being used in education for curriculum development, both formally and informally. Of course it's used by researchers, science researchers, federal external social scientists draw from it because so much of the application of it informs human behaviors. So there's all kinds of social science that is connected to this. So yeah, I mean we kept that really broad and that was baked into the to the data collections we did both qualitatively and through the surveys. Good okay I'm got my eye on the clock. Any further questions. Scott. I was just going to ask real quick you, you briefly mentioned the emergence of AI systems and how they might affect some of the flexibility or the ability to be responsive in the NCA moving forward. Especially in terms of thinking about this is essentially something could be a formative process that is built into how the NCA works long term. It seemed like you had more to think and say about AI systems and what they could potentially bring to the table. But maybe there's not enough time, we'll see. I would just make a brief comment which is, so I'm a active member of the American Evaluation Association. And there are many evaluators in this space that are touching upon climate issues and there's it's an active community. Within that we examine all kinds of questions around evaluation assessment, something that, you know, is absolutely being acted upon with the with the birth of chat gtp and gtp. So I have these questions around what it does for our jobs and our work, just like it is happening in many other spheres, education, etc. So we are definitely playing with it I haven't played with a whole lot but I was just asked by a client to play with that to think about like what it does to qualitative data. My initial assessment is it, it has a long way to go. The human component is really important, but for for crunching through a lot of data, if the inputs are right just like anything like high quality data input may lead to something that could be usable, and that's being looked at in the evaluation community. Hello, thank you very much for coming this morning and sharing your thoughts, both about what your evaluation did and I think shedding some very useful item, where we are going forward. So thanks a lot for coming. I wanted to turn, given that we have limited time to our guests from the US GCRP, including some of the representatives from the supporting agencies. Thank you so much. Awesome. Let me put you hand the hand the baton over to you since you've got a group of people to introduce to us. But welcome. Thank you. I'm very happy to be here and that was really interesting. So thank you, Heather. Okay, perfect. Okay, excellent. I'm going to give an overview here, and then turn turn to a panel of three experts who have been thinking, not just about NCA for NCA five but NCA 10 in our sustained assessment working group. I wanted to start off by noting that what I'm presenting in these slides are my own perspective my own opinion. I was laughing this morning thinking about coming to the National Academies doing a present presentation on my opinion and telling you anecdotes with small ends. So, but that's, that's where we are here. So this does not represent, you know, official federal steering committee findings or anything like that. Next slide please. Right off the bat. I'll note that this is my thinking about the different audiences of the National Climate Assessment so you can all feel very free to fight me on this. On the left are mandated audiences that come from the GC area itself, which we, you know, deliver the report to Congress and the president, but really in thinking about us GC RP's mission writ large. We are tasked with assisting the nation and the world and understanding global change. I kind of think of this bucket as decision makers, which helps me separate it a little bit from the bucket to the right, even though those are very blurry boundaries. And I, I put the people in the middle, a little bit more as in my mind kind of our primary audience that we're working towards. I think the federal agencies of the US GC RP and beyond. I think state local tribal territories are a huge user of the national climate assessments. And then there's sort of a very large bucket which I have also called practitioners for lack of a better word that I think are, you know, professionals who might be using NCA data in their work as adaptation specialists or water utility managers or hospital planners. We are seeing a little bit more interest from the private sector and financial sectors. And then I have sort of another category of people making decisions at perhaps a household or business level in terms of buying homes or making investments. In the third column here I have important users and I really want to underline the important just because they're not in the middle bucket does not mean they're they're not important. They're not in the middle of the report. In fact, we had a really huge number of comments come in for NCA five from the climate literacy and education community so they are very actively engaged. They, you know, engaged in making NCA five more useful to them because they see themselves as a as a major user of the report. So this category media. We see the assessments being cited in the media on pretty much a daily basis. And for example, last summer when we had those wildfires and Maui, a lot of the news articles were still citing NCA for which was, you know, at that point five years old. So I think hopefully now you're going to see them citing NCA five more and more often. Next slide please. This is also something I made up, which is a rather imperfect spectrum for how I think people might be using the national climate assessment. Starting on the left side, I would say are people who have heard of the assessment before. And I think that's improving, but I still interact with people fairly regularly who have never heard of the NCA before. I was just giving a presentation where someone came up to me afterwards and said they were incredibly happy but also very surprised that their tax dollars have been going to this since 1990. So that is, that is not uncommon. I think in the second row you have people who might be looking for an answer to a specific question or they're curious or they want to debate their relatives over Thanksgiving or they're looking for a quote. And then in the third bucket we have justifying in action. And I have to say, if I was guessing, I would say there's a lot of users in that bucket. A lot of people who are quoting NCA to say, this is why climate change is urgent. Therefore, we should do something about it, but not necessarily further to the right on the spectrum where they're actually using the information to understand or prioritize risks, given everyone's limited staff and budget and resources, what, what risks are of greatest concern, whether in my community or in my sector. Beyond that we have people in the second to last right bucket, who may be making cost decision benefits so looking at the range of scenarios we're presenting an NCA, figuring out, you know, maybe they want to look at a high and medium scenario or two and three degrees global warming or 2050 versus 2100 and then comparing how those different risks might impact them under the different scenarios compared to the cost effectiveness of those different scenarios. And then finally, I think all the way on the right is, you know, the folks who are how high do I build the seawall how deep do I dig this culvert. How many cooling stations do I place in my community and where do I place them looking for very specific data to inform their actions. And if I had to guess I would probably put fewest people in the the far right bucket. Next slide please. This is also imperfect but I tried to map a little bit of where I think some of those users might fall. I would certainly hope that we would have more and more users kind of pushed to the right in, in my opinion. I think, you know, some of this might even be a little bit wishful thinking for where I place them on there but, of course, a lot of users can span these different categories as well. We talked about researchers as being one of our user groups who might be looking at the NCA text to figure out where the greatest risks are. And then looking at things like our traceable accounts or our research gaps database to identify what new research to fund, or how to develop the research strategy. Next slide please. I'm going to go through the next two slides kind of quickly, because this is just another way to break down what I had on the previous slide, but more in table format, and providing just a few examples of how people in these different categories might be using the assessment. I want to point out that the three panelists following me, Aliza, Daryl and Dan are going to be speaking a little bit more specifically about three of those user groups, so I won't, I won't linger there too far. Next slide please. I'm also not going to read this slide, because there's a lot of text on there. These are very specific anecdotes broken out by audience that I wanted to put it on here so that you would have it for your reference. A lot of these anecdotes are still from NCA for of course NCA five has only been around a few months so far. I can tell you for NCA five, we've been doing an ongoing webinar series and have had more than, or around 7,000 participants already attending those webinars with seven more to go, and more than 6000 people have watched the recordings of the webinars which I think is kind of. I don't always amazed when people watch recorded webinars. And then about 3600 people listening to our podcast. I've briefed congressional staff as well as federal and state judges, and of course agencies so I have my trifecta of the three parts of government that I've briefed. And then, just a few more examples we have some economic authors who are going to be speaking at the federal Federal Reserve Bank in New York this summer, because they really want their staff to understand how the risks of climate change are going to affect the economy as well as some of their activities like micro prudential supervision and financial stability. And then I mentioned media. NCA five is still, you know, relatively new we've had 2400 mentions and 7.7 billion impressions online so far, including traditional media. And like NCA for we are seeing NCA five being cited almost daily in the media. If this is of course free, it's it's earned media it's free media but if it were paid it would be worth $192 million. Next slide please. Beyond who is using the national climate assessment, and how they are using it we don't actually have a good sense of why people might be using it, or why they might be using it instead of other resources or, you know, other types of climate information. A lot of the bold words on this slide have to do with our process, and have to do with the principles that drive our process for for making the assessments robust and authoritative and trustworthy and transparent. And of course it could be any or all of these reasons or none of these reasons at all. Next slide please. NCA is made up of a lot of different elements. A lot of different chapters and even within the chapters different kinds of elements and I sometimes refer to this as sort of an onion type model where if you are just looking for very top line takeaway messages. You might be looking only at the overview or only reading the key messages. If you're looking for more specific information about your sector or your region, you might be digging into the chapter text, using those figures. And if you, if you are a researcher and you're looking to go even further into the technical weeds, you might be looking at our traceable accounts or our metadata, or trying to recreate some of the data sets, or identifying research gaps for your own research. So, you can kind of imagine how some of those different user groups might track to the different elements of NCA and how they are using them. Next slide please. We also have many different products alongside NCA, and some of them are new to NCA five. So we have engagement throughout the development process, including public comment and public calls. You'll hear a little bit more from Aliza on that. We have the website itself, of course, in NCA five we have the Atlas and the art gallery which is new. We develop a series of downloadable materials, including two or three page handouts which I've, I'm told teachers really appreciate PowerPoint slides zip files of all the figures and for the first time, Spanish translated PDFs. And then we have our communication products, which, you know, things like press social media webinars, and this time podcasts and audio books. Next slide and this is my last slide. There is a lot of questions on this slide, not a lot of answers in my entire presentation, of course, but the last question here is who is not on any of those lists or tables from my previous slide, and who are we not reaching. I think we've done a little bit better job in NCA five of reaching some audiences. Namely, I think you could almost break these out into process like actions we've taken during the development process, like hosting youth dialogues or holding tribal consultations have reached those audiences. Some of it might be the content of the report and see if I have has a really strong environmental justice theme throughout the entire report so that's why we might be reaching those organizers a little bit better. So that's the products where we're translating the chapters into Spanish, which I'm hoping will help us reach some more Spanish speaking communities. I've lifted listed a few of my own ideas about the people in the broader environmental community that we might want to target for future NCA's. I often feel like we don't even have the full choir yet. So of course there's also the question of who we should be targeting outside the choir, and should we be targeting people outside the choir. And then lastly, I will swing back to where I started with, even if we aren't expanding our audience. Are there ways that we can push the current users a little bit further along that spectrum I made up so that we can move people not just from oh I've heard of NCA before to a little bit more about actually using the NCA in some of their decision making processes. So I think with that, I am handing it off to Elisa next. Great.  I will change my perch. Aaron, you can stay where you are. But that way I think it'll be easier for the people in the room to, to see all three of you. Yeah, that's all virtual. Okay, well in that case there's no point in moving around. Great. All right. Elisa. Hello. Lisa Luskin from USG CRP. Thank you for joining us. Oh, great. Thanks for having me. We can go straight to the next slide please. Great. Okay. I'm Elisa Lustig. I'm a senior manager on the national climate assessment team and I also coordinate the USG CRP sustained assessment working group. And I'm here to provide some perspectives on NCA engagement and to get into some of the specifics of our NCA five efforts. Next slide please. Okay, so I'm going to kind of speak to three different areas versus the who's engaging with the NCA I'll get into that a little bit more. Specifically about the NCA five processes we used for engagement. I'm going to talk about a project that we did a couple years ago, evaluating the NCA use in different state. And so while sub national level climate action plans. And then finally, I'm going to get into some of the lessons learned and ideas that we heard from our sustained assessment working group seminar series, which was done in 2022. All right, next slide. So I thought I'd start by just talking about the engagement opportunities that we had throughout NCA five and to give you some sense for what we're actually doing to try and reach people. So I'm going to go through chronologically. First, we put out our draft perspectives, which basically outlines the big scope for what the report is going to cover. We did that in summer 2020 and we got 105 comments for NCA five. I wish I had the NCA for numbers and I don't. But I do think it's interesting to try and track over time. The types of comments we got on the prospectus were things like, you know, add a chapter on X or better coverage of why geography. So you're really big picture scope level feedback that we got from from the public. Next, we had our zero order draft public comment. So that was an early 2022 almost I guess a year and a half later. And this was the first time we ever did such your order draft public comment for a national climate assessment. We paired that with a workshop series, which I'll get to in a minute. And we had for this over 900 comments just over, which was really great. The early stage feedback is critical because we want to make sure that whatever we're doing in the national climate assessment meets the needs of the people who use it. And so that's why, you know, that was at least a big part of the impetus for adding this early stage comment phase. We had tribal consultation formal tribal consultation also a first, I believe for the NCA that was in late 2022. And this was initiated originally by OSTP in an effort to develop guidance for the integration of indigenous knowledge across federal agencies, but we took the opportunity to join them. And, and had some framing questions and around the information quality guidance we give to authors and ways we could improve that to better represent indigenous knowledge. I think that was very successful because we completely revamped our guidance and that came part way through the process, but I think we heard some really positive feedback about it and the ways that authors felt more empowered to integrate indigenous knowledge given that input. And that updated guidance. So that's important for NCA 5, but I think it's going to also lay a really important foundation for NCA 6. Fourth, we had what Allison mentioned the Art by Climate Initiative. This was a first sort of experiment for the national climate assessment wherein we put out a call for art across the country in late 2022 into early 23. That was a call for visual art to be included directly in the national climate assessment. We did this kind of for two main reasons. The first was to kind of increase the community of people who were involved in the national climate assessment process. So obviously bringing in artists and making the report, perhaps more accessible to folks who wouldn't typically see themselves in a 2000 plus page scientific document. So, so bringing people to the table and then also making the science stick so making the issue or not just the science just the issue itself of climate change more resonant. And with a broader audience and to help people internalize it in a different way than they might perhaps through text and numbers and so forth. So that was quite successful. We had 850 plus submissions from across the country. Happy to talk more about it. If folks are interested at the same time also late 2022 early 2023. We did our classic third order draft public comment. We've done this now for at least two assessments, if not more. We're midstream through the process we released the draft report. It's of course concurrent with the NASA comment phase. We had over 3000 comments for NCA for just under for NCA 5 and I don't know, you know, why. In some years there's more engagement on public comment and some years there's less. But I do think that is an interesting question of why people choose to engage or not.   Certainly, certainly some received more comments than others, which is kind of an also an interesting question of what draws people in. I can think of a number of reasons why certain chapters may receive more coverage than others. And at that stage, unlike kind of back in the zero order draft phase, these comments were much more specific at this point we weren't really in the business of adding wholesale new key messages but we were in the business of finessing those key messages, adding case studies, and so forth. So we had more of a targeted review at the minstream phase. And then finally, we are currently hosting a webinar series that Allison mentioned, we have at least one for every chapter and we're also about to get into a set of regional workshops in person. I'll speak more to that in later slides. All right, let's move on please. Okay. I'm going to get into some of these in a bit more detail. This is the zero order draft public engagement workshop slide. So, like I said, this was early 2022 that we were doing these. We hosted 34 virtual workshops over six weeks, at least one per chapter. And we had for the Caribbean chapter we had one in Spanish and one in English. Oh no we had one in Spanish but we had simultaneous translation for the southeast we had to they did one in the evening for example to bring in folks who wouldn't be able to take off time during the work day. But for the most part we had one per chapter. They were long. So, these were between four and four and a half hours so it's a pretty intense. It's a burden you call it participation opportunity, but this was a more intense option for folks who wanted to engage this way and really get into conversation with authors about the about the content of the chapter. This ran in parallel with public comment, right, which is a much lesser intensity so I think that's interesting to think about is the types of engagement we ask of people and how the type of engagement you put out there might enable different folks to participate. So we like I said we had 34 of these. We had over 7000 registrations by 4000 individuals from all 50 states plus a number of territories. We had ultimately about 2800 people participate, which is a pretty good rate of a pretty good ratio of registration to participation. We facilitated these so we they were facilitated by our 450 plus authors and breakout rooms and then plenary different interactive modes. We at the same time received like I said over 900 written comments, and I broke down the types of people it's pretty rare that we actually know who is showing up to participate with us and that's limited by things like the paperwork reduction act which limit how we can really pull out to survey them. For example, when they register but in this case we made a concerted effort to make that happen. So we had pretty strong representation from government. I wish we had broken this down a bit more but government as a whole, about 41% and you know also pretty good from academia and nonprofit sector as well. So here on the right percent workshop registration by region. This isn't by size of the region so it's the unsurprising perhaps that the Northeast had the most with the Southeast together. And then you know, get smaller down towards Alaska and Pacific Islands region in the US Caribbean. Okay, next slide please. Okay, so that was what we were doing at the outset of this project. Now we're here at the end doing this post release webinars. So we have one webinar for every chapter except for the Caribbean chapter which did to one in Spanish one in English. The goal here is to share the report findings to the broadest possible audience. So this is really meant for anybody the general public to academics and decision makers themselves. The goal is to close the engagement loop. And I think that's true of all of the post engagement efforts that we do it's, you know, at the outset we really ask a lot of people in terms of providing input at the formative stages and we don't want to just take that input and then never be seen or heard from again we want to show up and show people what we did with their input. So really getting that full cycle in. Of course it is a cycle so I'm really hoping that we'll be able to also use some of these post release engagement opportunities to ask people what they want to see from the next one and keep that conversation going. Next. So within the within the context of the webinars. Yes, the authors are providing information on the key messages they're talking through their various figures and narrative texts. We have left a lot of time for Q&A almost 50% of the webinar which has been wildly successful. People have a lot of questions and we never get to all of them. We've had as of Monday, 6,584 people attend so far but actually as of yesterday we had another nearly 200 so we're on the up and up 28 webinars to date seven to go. So adaptation overview and water had far and away the most participants. And as Allison said, over 7000 people have watched the recorded webinars, which is awesome. From this I've drawn a number of lessons already. First of all that that Q&A time is really valuable people want to interact and people have questions about the assessment scope. They want certain figures and and things like that they have questions about why certain things were included or not. So the Q&A is really valuable. The next is that people really want to hear from us and I don't know if they really want to hear from the NCA specifically or GCRP as a whole but we seldom do these public facing webinars and it really makes me think we ought to think about doing more of them because they've been so, so widely, widely attended. Next, we need more Spanish language outreach as Allison said we're translating the entire NCA into Spanish for the first time. It's a first, and I think, you know, we had people show up to our Spanish language webinar but for the Caribbean chapter but I think we could have more people. And so I, you know, that's going to be a growing area in the future, I hope. Next, author networks matter. Perhaps you've heard of GCRP or in general this concept of network of networks, where in GCRP is really kind of the convener of people who have their own networks out across the climate space. The GCRP itself is not the sole holder of all stakeholder groups and I think that that really, really is brought to bear and how much the author networks are important for this. When the authors do spread the word about this about the assessment about our events about the opportunities for comment, people show up and I think it's really quite powerful. Next, as I mentioned before, it can be difficult to know who we're reaching, and that is true within the case of these webinars I know how many people were there but I don't know, you know, much about them. I'd love to know, you know, where people are coming from did we get anyone from some of the states we don't typically see much representation from. I think that would be really useful. And next, it's been really valuable to leverage the post release hype to bring people to the table when we release the report. People are very excited. We publish our entire webinar schedule upfront and we had people sign up wholesale for webinars from November through the end of March. So, I think that was that was a very strategic move and leveraging release time is can be really important. Okay. Next slide please. Public engagement workshops. So, we are about to launch these the goal here is to increase the exposure to an uptake of the assessment science and to offer a more substantive opportunity for discussion between authors and participants. These are not, these are in person and they're not intended for, you know, I mean, while the general public is welcome. These are much more targeted to practitioners to academics to decision makers, folks who are using the NCA science in their in their professional work. These are going to be very interactive in nature and context specific meaning that we're giving authors a lot of leeway and designing the work workshop they feel will best meet the need in the region. I'm going to skip over the timing of these we're about to publish it in the NCA newsletter. So, stay tuned for that. All right, next slide. Okay.  She looked at about 330 plans total. And search them all for the phrase national climate assessment or NCA finding that 25% of all plans reference the national climate assessment. She showed an increase in NCA references over time noting that none of the plans that came out of NCA one that came out after NCA one reference the assessment. Very few plans referenced it after NCA to saw an uptake after three and then an even bigger uptake in references to NCA for. So that is an interesting thing to note that we are perhaps gaining visibility and relevance over time. Next slide. So these are the types of references that came up in these action plans. First, just as Allison said, the NCA as justifying a call for action. Even New York City example here. Next, NCA as a source for definitions. And I hope that we'll see even more of that given the first ever national climate assessment glossary that we released. Next, as the NCA is an important place to contribute states encouraging their scientists and and others to participate in the national climate assessment. After that, NCA of course is a source for scientific information, whether that's basic science projections or information about impacts. And then finally, NCA as a guidepost for planning metrics. I love this one. The Tampa Bay region. Is developing a regionally consistent sea level rise planning scenario for the coming decades. And they require it to update every four years immediately following the release of the NCA. So those are the kinds of ways that the NCA is being referenced and used and through these climate action plans. I think it's a. This was a kind of nice example of a quantitative project we were able to take on that feels evaluation ask. Next slide please. Okay, in these last 2 slides, I want to talk about some of the findings from our sustained assessment working group seminar that we held in the year 2022. The working group. Thanks largely about was thinking largely about the future of the national climate assessment and we invited speakers like Richard Moss Naomi or us keys. Louis Rivers, Catherine, Hey, oh, folks were thinking intensely about assessment over time and the future of them. And this is some of the feedback that they provided relevant to engagement. First saying that the supply side model of scientific information does not work. We need to include stakeholders from the get go in the problem definition and framing phases. You could say we're sort of doing this through our Zod public comment for example, but I'm sure there's more we could be doing. Next, get rid of stop and start involvement, which will change our notion of engagement and think of building ongoing relationships where people can chime in at any time. That is interesting because it totally challenges the notion of draft comment, maybe there's a more sustained and continuous way we could be collecting feedback over time. Next, see assessment as part of the decision making process and plan accordingly. Actually, if we could go to the next slide. I think these few bullets here get into that a bit more. You have to understand how the assessment is being used in order to make it use to make the assessment itself useful. And so we need to understand decision maker motivations constraints resources cultural context, etc. We can also understand how the assessment fits in with users decision making process and design accordingly. So I think that these could be informative evaluation questions themselves. Perhaps not looking at specific NCA's but thinking about it more broadly. Next USG CRP as a magnetic field bringing together and guiding the assessment community that gets to the network of network concepts I mentioned earlier. And then finally, perhaps the biggest picture thinking one is the building of a climate extension service that would revolutionize how communities use science and how science hears from communities. So I think CRP gets more into the services space. I think this is a really exciting concept. It would really of course challenge our assessment current assessment model but I had to include it because I do think it's a very exciting vision for the next, let's say a couple of decade or so I don't know. So with that I will close out hopefully didn't take too much time and I will pass it over to Darrell. That was a very thought provoking and wide ranging discussion. Darrell, are you there. I am. Great. You are, boy, you're in a nice place. Thank you. Yes, that's my local lake. I miss shaking all your hands, but still happy to be able to participate. So, hi, I'm Darrell winner. I work for the environmental protection agency. I've served as EPA's federal steering committee representative for both NCA four and five. And I also co chair the US global change research programs sustained assessment working group that you just heard about some of our activities from Lisa. Like Allison, I have to tell you that my remarks represent my views and are not necessarily those of the next slide. Now, could you speak up a little bit? I'm having a little trouble hearing you. Okay, I will speak more loudly. Thank you. So I'm going to share a few more. I'm going to add a few things to Allison's and Lisa's remarks. We'll start. I wanted to give you some examples of the use of national climate assessments by federal agencies coming from EPA. I'm biased both in my knowledge and what I think is important. So my examples are highly specific. First, the 2009 endangerment finding for greenhouse gases that serves as the scientific foundation for all of EPA's policies and actions regarding climate change. It actually references the second national climate assessment. In 2016, EPA published a separate endangerment finding for aircraft to move forward on policies regarding greenhouse gas emissions there. And that cited the third national climate assessment. The fourth national climate assessment was cited in the recently released climate change adaptation implementation plan for our office of air and radiation. All of the major offices in EPA have an adaptation implementation plan. And I would guess that most of them cite NCA for and then tomorrow. A new round of adaptation plans from all the agencies are due and we expect to find a lot of references to NCA 5. Next slide. The national climate assessments and other USG CRP assessments that are also very important like the state of the carbon cycle report and the climate and health assessment. Definitely inform agency research priorities. Folks in USG CRP compiled a really nice research gaps database that looked at 6 different climate assessments. Both volumes of NCA 4, NCA 3, the carbon cycle report, climate health assessment and a few others. And you can actually access that research gaps database based online and look at their really nice work of putting keywords and tagging across all these assessments and seeing what the research gaps look like. As Alisa mentioned, one of the things we did in NCA 5 was to have a glossary. And we think this will really be helpful to help lead towards consistent terminology and use of terminology. And then as we're seeing new agencies joining USG CRP, which we're really excited about. We're hoping that we'll see use of the national climate assessments in their strategic plans. Next slide. So, as we think about evaluation of NCA 5 and using that to inform NCA 6. I think about what could we be doing either in terms of the format or function of the NCA, the content and our outreach and engagement to make NCA more useful to people in our many user communities. Should it be shorter? It is quite a large reports having the web being the official report makes it not quite as apparent the height of the stack of the thousands of pages, but they're still there. So, there's always a push to try to contain the ever expanding amount of pages. At the same time, people want to see us cover more areas, more topics. And so that naturally pushes us towards a longer report with more chapters. What tools should we use? What formats work best? Questions like website use, user experience, our podcasts and videos. How can we best get our messages out? And then how can we use climate scenarios to communicate future climate change? Traditionally, the larger community has used the many time based pathways like the RCPs, but we've also seen people explore the use and utility of global warming levels. So, what the world might look like at, you know, 1.5 degrees Celsius, which we're kind of already at two degrees, three degrees, etc. provides a different sort of framing and potentially a way to compare more readily across those different scenarios. Next slide. So Allison gave me a prompt that as a federal steering committee member, what would be most useful in an NCA evaluation that would inform the decisions the steering committee faces. So my response to that are what forms would maximize the utilities of the NCA to the multiple user communities. I struggled a bit as I wrote that sentence started mainly with singulars, but kept adding plurals. And that really is in a nutshell, highlights a lot of the issues we wrestle with as a federal steering committee. It's not a singular form. It's many forms. It's multiple user communities. We have in the past and it would not surprise me as we go forward to see conflicting answers based on people's different perspectives across the many user communities. Of the national climate assessment. Are there ways that we can better integrate user communities into the production of NCA 6 and beyond. You've heard both Allison and Alisa lightly touch on this as a federal government entity. We do have rules and regulations. We have to follow 1 of which regards the collection of information limits. Interactive surveys without. Difficult permission to get to go beyond that limits. Connections to communities as well that kick us into a different realm of. Of rules that govern those sorts of relationships. And then 1 of the key components of the. And finally, and maybe most difficulty as we rate way or resources, mainly people's time. How can we prioritize certain user communities over others? We don't usually enter that questions, but at least in the 2 NCA is I've been involved in we land at that place. We can't do everything for everyone. And so who do we prioritize? I am seeing a message about my connection. So I'm hoping you're. Next slide, if you're hearing me. And I'm going to hand it off to my SOG coach, Eric Dan Berry, take it away. Thank you very much, Darryl. Great. Can you see and hear me? Yes. Perfect. Thank you. Yep. So I'm, I'm Dan Berry. I work at NOAA and the climate program office and served on the federal steering committee for NCA 5 with Darryl and also coach here, the SOG with Darryl. Hard to follow up those 3. And I guess I'll just say that I co-signed everything they said during their presentations, but I'll try to add a few additional little bits of information and perspective to the fantastic points that have already been made. If you could go to the next slide, that would be great. Thank you. Yep. So a few things. First of all, on the structure. This question of the national versus regional chapter format of the assessment. The global change research act is very brief as I think all of us are aware in terms of what it requires of the production of the assessment in terms of what it's expecting to see in the assessment. And in terms of instructions for carrying it out. And over the years, the assessment has flourished and grown and includes many topics beyond the small subset of topics that are included in the language of the act. And also these regional chapters, which are not called out at all in the act, although are, you know, a very reasonable interpretation of trying to communicate about impacts to the public. And so, you know, in an evaluation, I think we really fundamentally want to have a sense of the chapters and how suitable they are for the audiences that that we're targeting with the report. And is there a good balance in breadth of topics covered in the report are the chapters responding to contemporary interest? And I think, you know, there have been examples given where we've added chapters and also try to bring different types of expertise into chapters across the report to really address some of that need. And then in terms of this utility issue for federal agencies are the chapters effectively helping the agencies consolidate their knowledge into the assessment and represent the work that we are doing across the federal government. I think that's another question too, as far as performing an evaluation of the utility and performance of the report from the perspective of the agencies. And then this other question as well, you know, I think I'm going to hit on this a few times in my comments around exactly what the assessment is in the broader ecosystem of climate services and and as a tool that's useful for various climate action communities, which is the fact that the assessment serves as a really powerful social tool. So, Elise has spent some time, you know, detailing the massive amount of outreach that's done and the thousands of people who are engaged through that. And the many thousands beyond that who interact with the report directly through the website and other materials. But this is, you know, at some at some level, not just an assessment of literature or impacts. It's a it's a convening tool and it serves a really important role in that space. And I think evaluating the way in which the chapters, the structure of the report, the various elements of construction of the report, like the workshops and outreach activities, how those amplify the social role that the report plays in the broader climate community. I think that's very important for the regional chapters. I think there's an enduring question about granularity in the report and usually communities that are taking action really want very high resolution or local scale information. And in many ways, that's beyond the breach of what the report can do as a more general assessment of climate impacts and having funding and scale to really operate at a regional scale with some state level information. So these issues of granularity and really evaluating is the report still serving communities properly at the level of granularity that it's able to provide at its current scale and scope. And the regional chapters are an effort to try to get down to that that sense of scale. So I think that's an important issue to think about in the evaluation. This next bullet on public interest in economic social science and culture heritage. The assessment, you know, has historically focused on primarily physical climate impacts, but increasingly over the last few years has really considered the way in which communities interact with those risks and the social context and understanding of climate impacts. And it's something in particular, I think that, you know, Allison, you know, much credit to her and Alisa with the social science coordinating committee at USG CRP really thought about this between NCA four and five. We received a lot of feedback indicating that there was a desire to see more social science engagement and perspective being brought into the individual chapters. Not just adding an additional chapter that touches on prominent social science issues, but really integrating that type of knowledge and framing around climate impacts across the report. And I think we were successful generally in that respect that that's something that would be great to have a benchmark in an evaluation process is do communities that are interested in the social dimension side of the way that we build resilience or understand the adaptive capacity of social systems, how communities perceive and deal with risk. These very important social science concepts that are key in their interaction with climate impacts, really bringing that across the report. And did we do that effectively with NCA five? What's the understanding of that through trying to understand that through an evaluation process. The next slide, please. On this slide, I'm going to focus on the concept of climate services more broadly. This is a little parochial to Noah, but the federal government at large as a provider of climate services to the public. Something that I don't think is clearly established and would be really great to try to elicit through the evaluation process is the role that the assessment plays in climate services nationally. And what its perception is amongst different users of the report as far as whether it itself is considered to be a climate service. I think at least Allison provided provided a framework for this. Indicating across that cascade of different categorical user types that our hope is to move the reports use a bit to the right on that framework. And that right side is really, you know, at the very highest precision end of climate services. So understanding the extent to which the report is seen as being a climate service functions as a climate service. I think it's very important. And, you know, I would say from a Noah perspective that we generally view the report as sort of a lower case. So operational output of much of the climate work that we do in the agency. And so I think it's reasonable to argue that that it is a service in some form, but how does it then fit into the broader landscape of services. And evaluation I think could really elicit that perspective for us better and create a sense of the hierarchy of climate services and how the NCA fits into it is, is it a cornerstone of the building. Is it more essential than that is it more flexible than that. And then from from there, we can really think about how to grow our climate service activities around around the assessment. In terms of the tools, the data and information, the on the ground stakeholder engagement activities. Does that grow directly next to the assessment in the federal landscape or at Noah specifically? Or is it something that kind of remains in parallel in the two processes speak to each other. So that's another question that I think would be wonderful to sort of arrive at some perspective through evaluation. Next slide please. This is my company and list to work Darrell presented just a selection of different questions that are somewhat random, but that I'm thinking about as a steering committee member. First of all, again, going back to the GCRA Congress is one of the essential recipients of the report named explicitly in the GCRA language. And Allison mentioned this already that she's had some briefings and whatnot with congressional staff, but I think better understanding if the report is serving the needs of the explicitly named recipients in the GCRA language is very important. And it's not always easy to get to that. So an evaluation process should look at that. The report structure Darrell touched on this a bit. We're, you know, interested in in the structure in how the structure might vary over time, whether it needs to evolve. This is the third cycle where we've delivered an assessment that looks somewhat similar in in feel and structure and appearance with some very significant changes in content, of course. But is this structure still serving communities? Well, I think we should really try to understand and benchmark that over time. And this third bullet kind of goes to that point as well where, you know, right now the report is very linear. It's in chapter format. There's discrete topics distributed across chapters. Is that serving user needs as well as we need it to do? We need to consider either reshaping the report or bringing in tools such as AI to help individuals query the report and perhaps draw information from across the chapters and make it a little bit less stuff typed in the way that the information is provided or presented. This fourth bullet, I think I've touched on already largely, but just understanding generally the role that the report plays in the broader landscape of climate service and trying to understand climate impacts and the information that stakeholder and response communities draw from it and other sources. I think the fifth bullet is largely similar to that point as well. The sixth point, you know, we haven't touched much on this, but a really wonderful aspect of NCA5 has been the art, the climate art that was contributed by young people and professional artists from across the country. It's really an incredible part of the report and, you know, many much recognition to Allison and Elisa for really spearheading that. I've heard from many individuals that the artistic aspect of the report really has drawn them in to it, which is part of the purpose for having included that in the report. And thanks to Elisa for sharing the link. Definitely go take a look at the art if you haven't already. I think we should evaluate how that has worked and if it has drawn in other communities, how it has altered understanding of the science in addition to being an entry point for folks who maybe are not going to show up to the report for the scientific and analytical content. That's there. It'd be great to understand that. And then finally, you know, really also looking at the federal use of the report and the role that the report plays for different federal agencies. I think there's probably a lot of heterogeneity there for NOAA. There are a number of chapters that can prominently feature NOAA's work in the climate space. The coastal effects chapter, the ocean and marine resources chapter give us opportunities to express our very specific mission space responsibilities at NOAA. The regional chapters as a highly regionalized agency are very important for our experts to have a platform for us to share regional knowledge. But I think every agency probably has a number of different incentives and interests in the report. And I think that in the evaluation, it would be great to elicit what those incentives are and interests and then how well the different agency stakeholders feel the report is performing to those interests. So hopefully those are some useful additional pieces of content or thoughts and thanks for including me on the panel. Thanks very much, Dan. Thanks for the whole to the whole panel, as well as Allison for the this overview. I think that I come away with it with a sense of the of NCA five as a much more varied and complex system than than just a report. What we do with that perception I'm not yet sure about so we'll talk about questions. 20, 25 minutes to talk with USG CRP people, please begin and I will try to monitor the people who are online, but Jessica to begin. Thank you so much for for those presentations and I'm reminded, Heather encouraged us to think through, you know, the utilization and also to prioritize and so in that spirit, I want to go back to the slide that you had Allison where you showed all those examples of use cases. And what I'm trying to wrap my head around is what would be most useful to you and your colleagues. Is it going into depth with those examples to be able to understand some of those questions about well why did you use it and how did you come across it and what would have made it more useful so really doing almost case studies of a limited number of uses. Is it trying to capture the landscape up here are, you know, all the different one example from each field about all the very very many different ways this could be used, or is it trying to do something more quantitative like the work that your fellow did to try to say well we want to give a number to show impact that this is the number of times it is doing there so if we can't do all three where do you fall in terms of you know what would be useful to you. In depth, totally wide or something kind of quantitative to have that punch in terms of the impact. I want to see if Dan and Darrell have a reaction to that first. My first reaction is to appreciate that you're feeling our pain and just trying to evaluate this. We struggle with many of these things issues in planning and producing national climate assessment my personal off the cuff take would be an in depth evaluation on maybe five to less than 10 sort of important use cases would be of most interest to us in thinking about how we structure form and function of future reports for those important use cases, but I'm curious to hear Allison's perspective if sort of a broader sweep of everything, how that might compare in priority. Sure I'll go and we'll see if Dan has an answer. I was actually going to say the same thing Darrell so we're aligned. I think the in depth. The in depth one appeals to me most because I think if we could get some of the, some of the people that I was talking about who are kind of in the choir, using it more. I think we can get at that notion that Aliza was talking about of being the network of networks. So I don't know that USG CRP directly is going to go out and find a lot more, you know, completely new users. But if if we're doing a better job of reaching the people who are doing that. I think that would, that would probably be the best bang for our buck on this so. Yeah, I'm going to vote for the in depth. Dan, my thoughts. I'm not going to add much I would also vote for the in depth stuff. I agree with your summation of it. Matt. Hi, so the previous comment was from represented from planetary health alliance. I'm in a division of climate medicine. You've worked really closely with the American geophysical unions geo health section and as you all know there's a really strong interest now in bringing together varied communities of practice on the public health side and on the geosciences side. And to my mind, one of the great services of these climate assessments is pointing out what kinds of expertise should belong in a transdisciplinary conversation. And I'm wondering if we did focus on more of a narrow case studies thing like you had pointed out nurses as one group. How a nurse uses this information might be very different than how an epidemiologist does, but they're both within this planetary health space. So, it's just a suggestion to maybe not throw the baby out with the bathwater and also mentioned the wide range of use cases before getting into the nitty gritty of specific examples. Yes, following on this idea of trying to identify a more limited number of user groups and do the deeper dive. I guess my question is, how to identify which groups to do the deeper dive on. And when I really found it very interesting your, you know, your schematic there where you showed the different, you know, types of engagement that people had and I think you said, you're trying to get people to move more toward the right. On the other hand, you also said that the very far right was a very small part of the population. So the question is, do you say, oh, that's a small part of the population so we won't worry about that. Or do you say, that's where we want to get people to so how so we want to include that group. I'm just trying to think about how one would make the decisions about what groups to prioritize. If you have any thoughts on that. Yeah, my thought is kind of a triage sort of situation. I think the federal agencies so. So first I'll say, in my opinion, I would be aiming at that middle box of the decision makers. And, and within that middle box of the decision makers. I think we're doing a great job already of the federal agencies. So, maybe not focusing on them. But I think where I'd love to see I'm not impressed by the 25% number of state climate action plans who are setting the assessment. I would love to see it used much more at a state local level, and especially as we're thinking about how to evolve these assessments with the regional chapters without the regional chapters with different forms of regional chapters, having more awareness at the state local level of the national climate assessment. It's a really helpful resource. It is like the sweet spot for me. And then I think within that broader bucket of decision makers, and speaking to Matthews point to I think we might have to pick and choose what I mean there's, it's such a broad range there's, you know, engineers and nurses and completely different and then as you said, a lot of diversity within those communities. So I'd probably be picking a few of those specific ones. And then I would say, you know, on the other side of the triage. Things like individual households or general public, or no, I maybe wouldn't fight in there yet. I don't know that the national climate assessment has to be a household name just yet. So maybe that's how I would triage the kind of the middle ground. I would add. Can I add to what Allison said their time. Yeah, please do. So thinking about that hierarchy, the framework that that you mentioned Kathy that Allison presented. And as this linear thing that has a bunch of chapters and it's presented in that format and is operating at a regionalized scale at its finest. It's going to struggle to inform actions. You know, it's working with a wide array of communities like, you know, Allison calls out in her quote my building specifications are blah, blah, blah, blah. The engineers want analyze data sets that are very specific and have specific analytical context around them and thresholds and things like that. And they want to co develop that they don't just want to receive it. So I think something like the assessment in its current state is going to be difficult to ever get all the way to the right. But it goes back to this question that I'm wondering about about how the assessment figures in the larger landscape about of federal climate service information. And one of the interesting things is, and I've been managing this at NOAA, NOAA provides the funding for the technical support unit that does a lot of the work on the assessment editorial work scientific work developing data sets for authors and then the production of the report and that team has 20 people ish who are involved with their good writers. They're good. You know, they know a lot about the science. They are excellent with data analysis and web tool development. So there's a wide array of skills on that team. And for me, as a manager who has been trying to build initiatives around climate data services projection services for stakeholders like engineering commit communities or reinsurance. The assessment team has actually been sort of a central kernel of how we're trying to build this in a larger scale. And so it's kind of interesting to me to see that when I look around NOAA at where, you know, the climate service activities can can really flourish. Right in the team that produces the assessment. I think that speaks to the to the to the model in my mind that the assessment is really the central thing that a lot of other activities can be built off of in both a practical manner. But also in terms of like a broader considerations manner in terms of the, the kind of social mechanism of the assessment more broadly. So, I don't know, you know, exactly what the NCA itself the full extent of how far it can get to the right on on Allison's framework, but I think it's worth considering that that it is, it is a very useful platform on which to build additional services, and to extend beyond, you know, this really important convening role and informational foundational information setting role that the assessment already plays so well. So I want to chime in. Sure yeah. Father comes to my mind is similar certainly along these lines but thinking about the compelling stories that connect this, the NCA information, not only to decisions of all types but to actual on the ground. Change let's just take adaptation and all the different kinds of decisions that connect to that. We look at that very general concept into two main areas of things that we can always iterate on and improve upon versus those that are foundational that we make decisions on and it might we might need to live with it for another 30 to 50 years infrastructure for instance or moving communities. Things that are more durable that we actually are already having to decide on now. And I would include education in that. So educating the next generation of people who are going to help us move this into a better place, ideally. And so, in some senses, that's a durable tangible thing happening. I think those kinds of I mean depending on who the audience is but say it's Congress. The stories sometimes are used to compel funding of course among other reasons but connecting it to on the ground is actually not I would argue not too far to go it seems kind of overwhelming at times to think about connecting it all the way there. But I do think there are stories there from what you all have laid out about who's using it for what, and maybe one avenue is to think. Would that be with that rise to the level now of saying, this is already compelling. Important decisions that are going to endure in our landscape in our end in society for the next few decades. I just had a quick question. I haven't had a chance to look at some of those out facing products on some of the websites, but I was wondering if your team had talked about opportunities for customization or personalization. Now, I know that there was a graphic where they were those regional workshops, but are there any tools like for example, you know, someone can put in their zip code and it'll spit out some information because I know a lot of community groups. They just want information and they're happy to sort of, you know, analyze that information, but it's great to have that unit of level analysis information. So, I'm just wondering if I'm probably guessing there's a lot of resources and money involved in doing this, but I'm just wondering, have you had those conversations and what those conversations look like. I'll note that we have the NCA atlas, which is atlas.globalchange.gov. And that is using the NCA five data, but in an interactive mapping tool, kind of speaking to the services piece that Dan was talking about. And that lets you choose the variables of interest, choose the global warming level of interest, and then zoom really far into your state or county to create a map. You could overlay the Justice 40, you know, layers over it. So, so really customizable maps for those purposes. So if the exact figure you need is not one of the 400 figures in the report, you can create it. I would say that's the major one that we have now, but a conversation that we have a lot across USGCRP is developing derivative products. We have all these years, all this time, all this money creating this giant resource of wealth, wealth of information, 2000 pages of it. But if you could distill that into a six page brochure for a coastal fisheries manager in the Northeast, that would be, you know, that information is probably in there. And if you could tailor it in that sort of bespoke way, it would be very effective for that audience. We have ever really successfully created derivative products like that. And I think we would have to be really thoughtful about, you know, who were prioritizing in those audiences, and we would probably need more help from agencies to sort of champion those projects. Scott. So I know I brought this up before in our committee meetings but it's great to have this audience here to talk about it with. So, in the spirit of being a network of networks. Elizabeth mentioned, sort of a national climate extension services network. And obviously the federal government has already spent a lot of money creating networks like the caste system in the Capri says and things like that. In the USDA climate hubs. I'm just curious thinking across different government agencies. Is this something that you're thinking about and engaging those other existing climate services organizations throughout the country and, you know, what are your, what is your thinking about that and any conversations that you've had. I'll say briefly that. So USG CRP is just beginning to think more about what it would look like to expand into a services space and involved in that are all of the organizations you just mentioned and then some we it's like everywhere you look you realize that agencies have regional entities doing that outside of just a headquarters right like you mentioned those three and those are the three we start with but we have the BIA tribal liaisons, we have the EPA regional offices. We have the DOE has all these regional entities and we have the regional climate service directors from NOAA the RCC regional climate centers from NOAA. It's like the list goes on and on there's so many people doing regional work already and we have through the national climate assessment. It's a convening force right so we brought folks together around the NCA to try and think about ways that we can best model the report to meet the needs that they know their stakeholders have and then include them in our process as well. So we've done some of that but I do think there's a lot more we could do in thinking about how to coordinate efforts and also very much respect that they each have their own missions and goals and governance models, funding structures, etc. So it's complicated to do the coordination work but I think especially as we think more about the assessment to services grayscale, if you will. Figuring out how did, yeah, do the coordination that's going to become more and more relevant. Carlos and then I want to wrap up with a question from somebody we haven't heard from yet but go ahead Carlos. Thank you. I have a quick question for for Alison because I see that the universal users of the NCA is so big and so complex that what is the main objective of the assessment for you. That's a tough question and kind of a personal one to it. What I want the assessment to do is move the needle on change in the US in terms of mitigation and adaptation. So, getting to what I was talking about a sort of those causal outcomes. I don't, I don't want people, I don't want to do the assessment just so people have heard of the assessment. I wanted to actually inform those decisions in a way that makes it easier to do those decisions and that we learn from them, and that community see what worked and what didn't work and then build and build and build.  Okay, now, we're running at a time and Mike Cooper books been participating on online but I, we haven't heard from him yet. Now, a lot of good things have been said, and confidence been offered to Allison. And I'm sure, and she is certainly deserving of them path breaking things like the, by climate initiative and so on. She's going back to her agency.  Any other thoughts about we spent a lot of time recently talking about the evolution or emergence of a national climate service ecosystem. I guess the question I put to you is, do you see the US GCRP as an arena in which questions about what is feasible to do in climate services, and, and how to build the coalition within the federal including Congress for climate services is US GCRP the focal point of that kind of conversation and should it be. So, Kai, thanks for the for a moment I'll try to be very brief. The answer is yes, that that's a very brief answer. Yes, US GCRP is the venue for the coordination of climate services across the federal government. Committee on environment chartered a new subcommittee, that new subcommittee will have its first meeting late in March. There's a lot of ground work being done right now the new subcommittee is called the subcommittee on climate services. A lot of ground works going on right now on who will sit on the subcommittee once responsibilities will be how relate to the subcommittee on global change research. It will relate to larger where it will sit within the larger US global change research program, so lots of lots of work being done right now that will be able to talk about, once it's all nailed down will have have all the details, but absolutely. There's a need for the coordination of climate services I think was generally agreed there was a fast track action committee that put out a report that endorsed that, and that that report also endorsed the fact that US GCRP should be holding the venue for the coordination of climate services. So we're very excited about that. And we see the national climate assessment as a climate service. I mean, it certainly is the sort of thing one would do.  Right, thank you because where this relates to what we are doing in this committee is that the evaluation of looking looking backward toward a nca five. In some ways, can click, we hope might clarify the discussion about where you're going not only with nca six, but with this discussion about climate services. What's been floating in my mind that as I've listened to this fascinating discussion is the question of bandwidth. If you look back at how macroeconomic policy took shape in the United States in 1946. The Congress passed what created and created the Council of Economic Advisers in the White House, and they issue an annual report, the economic report to the president every year, and this is a compilation of the economic statistics, the labor industry, and then they have a group, you know, special chapters in each report that focus on particular topics. And that's a, that's a volume that is a reference volume in terms of the data that it presents. And so it's a, it's a discussion, sending it, you know, the process, but it reaches directly a very narrow audience. It's a very important audience members of the Federal Reserve Congress congressional committees and so on, but their bandwidth is very closely described. And in a way you could read the GCIA that way, you could say, Okay, what we're, what we're trying to do is to create an annual report or annual report on the state of the climate. And then hand that off to decision makers across the government. Another way of thinking about bandwidth is the way that Dan was talking about and MOLA and the emergence really over a long period of time in a set of independent decisions of the creation of weather forecasting and the weather channel and the AccuWeather and, you know, there are a lot of private sector actors in this. And the way in which people now think about their relationship to weather conditions, I think in a very different way than, than 30 or 40 years ago. I mean, this morning, I was trying to figure out when to walk here to avoid the rainstorm. And, and I had, and I had at my disposal, a radar map and radar map with the projection of rainfall, so that I could try to choose the optimal time to walk from the hotel to this building. Now that's the kind of use, okay, so I mean, talk about a granular use, right, that's a granular use of weather data that was just simply certainly out of my glass, not very long ago, but now it's routine, you know, so what I'm getting at here is that the weather service and MOLA provide weather information across a very broad spectrum and they have to think in terms of a sophisticated system of developing collateral products in partnership with the private sector and so on. And, and that's a very different vision of what a national climate information production system would look like. And, and, and that's, you know, those are, and where you where the country will land with climate information I think is, is, is what Montia says, GCRP is going to be the arena where some of those questions are surfaced over the next several years, is that correct, Mike? Absolutely. I just want to be crystal clear. USG CRP is not going to be producing climate services. National climate assessment as a special case. The USG CRP the intention is, or even a nexus for the coordination of, of developing and providing climate services across the government and probably beyond the government. Right, and the Council of Economic Advisers, you know, which produces the economic report to t