Excellent. Well, I know from the few conversations I got involved in that there are a lot of really useful connections made at lunch and a lot of discussions and I trust that we'll have some time to connect at the next break as well and afterwards and that people will value the contact information that they've gotten from others here. This session is titled New Approaches to Unmet Needs Communication for the Whole Community and it has a couple of different aspects to it. So we're going to start first with a presentation by Wendy Brundabrund who's at the University of Southern California and she is going to talk about a topic that is near and dear to all of our hearts I think is how to be clear and improve risk communications and we will follow her talk with a short Q&A on her talk and then we'll move to our next two speakers. The goals of this session are to identify current and emerging strategies, barriers and challenges for communicating uncertainty and probabilistic information about tropical cyclones and to highlight unmet needs in communities at risk from tropical cyclones and potential solutions to meet those needs in the context of communication. Wendy are you there? I see Wendy's slide. Can you hear me Anne? Yes and now I can see you. Take it away. Thank you so much. Thank you for having me at this workshop. I'm really sorry I couldn't be there in person. I had teaching obligations but I'm happy to join online. So my name is Wendy Brundabrund. I'm a Provost Professor of Public Policy Psychology and Behavioral Science at the University of Southern California and in my research I study how people perceive health risks and environmental risks and I also study how to best design messages and interventions to inform the decisions of practitioners and policy makers as well as of people themselves to improve protection against risks. And in research on risk communication it has been found that if you want to help people to protect against risk it's important to be correct and to be clear among other things. And it's my experience that experts often work very hard on getting the science right so they're really worried about making sure that their messages are correct. But message wording is often an afterthought and so when experts design messages they have a tendency to use complex language with which they're themselves are very comfortable and that complex language is useful for communications between experts in their domain but they're not necessarily it's not necessarily the best language for communications with people outside of their domain and for members of the general public. And before I continue I want to highlight that there is a social science method for improving effective messages and it starts by identifying your recommendations but also it includes finding out why people may not follow those recommendations so you can do interviews and surveys with your target audience. Then we design messages to address why people may not follow recommendations and the message design is based on our findings as well as the literature that suggests what works and what doesn't work. And then finally it's important to test messages so we randomly assign people to different messages to find out which ones are better at improving understanding of risk and helping people to protect themselves. And of course it's important to use this process to design messages in advance of a crisis so that messages are ready to go when the crisis hits and you have some confidence that those messages will be effective with your audience. Now experts in many domains often use technical terms and I wanted to give an example of the problems with that from a project that I worked on with the United Nations Foundation. This project looked at climate change communications. As you may or may not know the United Nations Foundation convenes the IPCC the Intergovernmental Panel on Climate Change and they communicate about climate science to policymakers, practitioners and members of general public around the world. We asked the climate scientists who were working on those reports to identify key terminology that they thought would be central to understanding climate change and climate change communications and you can see those terms on this screen. Today I'm going to talk about two of those terms because they're central to behavior change. The first one is mitigation and when climate scientists use mitigation they are referring to the things that you can do to reduce your impact on climate change. And then the second term is adaptation and that refers to the things that you can do to protect yourself against the climate change that is already happening. So the climate scientists identified those terms and then what we did is we interviewed members of the general public in the United States about these terms. So we asked them how easy are those terms to understand and can you define those terms in your own word? And I want to start by a quote from one interviewee that sort of reflects the general sentiment in those interviews. One interviewee said they're talking way over people's heads. Now let's dig a little deeper so we asked people about mitigation and we asked them to rate it on a scale from one to five with one meaning not very easy and five meaning very easy. The average rating was 2.5 which is below the midpoint of three suggesting it was perceived as not that easy to understand. And then when we asked people to define mitigation in their own words they often struggled and one example is that somebody said oh mitigation isn't that helping to resolve a conflict so they confused mitigation with mediation. Then for adaptation it was rated as relatively easy to understand but just because people think a term is easy to understand doesn't mean that they define that term in the same way that the experts do. And so for adaptation a common definition that we heard is oh adaptation that's easy to understand I know what that means it means turning a book into a movie. And of course that is what adaptation means in everyday language but it's not what it means in the context of climate science. And so maybe it's not a good idea to use these technical terms because people may struggle with them and interpret them in a way that makes sense in their day-to-day lives. So we also asked interviewees to suggest simple wording to talk about the terms and people suggested some ways to talk about these words. So for mitigation they suggested you could talk about it by saying actions we can take to stop climate change from getting worse. And for adaptation they suggested well you can talk about that by saying actions we can take to protect against climate change. So it is possible to talk about these complex topics in the simple way. And then you might ask well are the words we use to talk about tropical cyclones clear. Well I could not find systematic wording experiments but maybe I missed some. I did find some suggestions that there are terms that may be difficult for people to understand. One example is shelter in place. So when experts say shelter in place they tend to mean that you should stay inside and not expose yourself to whatever risk is outside. It turns out that some people misinterpret the term shelter in place to mean that they should go outside and find the nearest public shelter and they may not know where it is so that means they'll be looking around for it. And so perhaps it's better to not say shelter in place but instead say that people should stay inside. Storm surge is also a term that may be confusing and people don't necessarily mean why they're at risk for it if they know what it is. There seems to be some confusion between terms like hurricane watch and hurricane warning or tornado watch and tornado warning and then there are a number of studies that suggest that people are confused about the cone of uncertainty and that inclusion of the cone of uncertainty in a message may draw attention away from recommendations about how to protect oneself and there may be other terms that haven't been tested. So what is wrong with using complex language just to reiterate? Complex language can be hard for people to understand. The average US adult reads at the seventh or eighth grade reading level and experts tend to communicate at the university level. Also complex language is generally not liked. Even people who read very well tend to prefer simple clear messages and complex language can be so off-putting that people don't even want to interact with complex messages and complex messages may not work and they can put people's lives at risk. So how do you simplify your messages? Well the general recommendation from research on risk communication is that messages will be easier to understand if they are written at the grade seven reading level or lower. There are various websites where you can paste your text message text into the website and it will tell you the reading level at which you've written it. There are statistics such as the Flesch Kinkade readability statistic that will tell you the readability of your text. Also it's important to avoid jargon though I realize if you're an expert in a domain you may not always recognize when you're using jargon. So another rule of thumb is to use short words. So words of one or two syllables tend to be more commonly used in everyday language and more widely understood. Say use instead of utilize for example and then also use short sentences. Of course simple language may still lead to some confusions if it doesn't address what people still need to know. So it's also important to get feedback from members of the intended audience so you can do interviews and surveys as well as randomized experiments to find out what works best for promoting understanding and behavior change and social scientists can help because you know social scientists may not be experts in tropical cyclones or in climate change but social scientists like myself have extensive training in doing interviews and surveys to find out how people perceive risks and also in designing and testing messages to find out which messages might be most effective at promoting understanding and helping to implement behaviors that protect people against the risk. So thank you very much for your attention. My contact information is on the slide and I look forward to your questions. Thank you Wendy. So we'll take questions from the room. Yes. Thanks Wendy that was a great presentation and it echoes a lot of what we've been talking about at the workshop about jargon. So in talking with experts though sometimes at least the ones I've talked to so this is completely anecdotally but they feel that using these jargon or technical terms gives themselves a degree of credibility so it's a cue that they're an expert in something and they feel that's important to convey for people heeding their recommendations. So I'm wondering what you would tell those experts who want to use technical terms to convey their expertise. Yeah thank you for bringing that up. There's some experts in various domains may have some resistance to using simple wording because they want to sound like an expert and they feel like their complex language is the right language to use and simple language is not exactly what they mean but the thing is if your point is to communicate to improve understanding in a particular community you need to use the words of that community and but so I tend to try to convince them by citing the research on that but generally I find what is the most effective is actually doing interviews. Doing interviews with their intended audience to find out how those preferred expert terms are interpreted by that audience because then you can actually show the confusion that people have and so that's why I worked with the United Nations Foundation on Climate Change terminology because they found some resistance among climate scientists in terms of using simpler wording and so if you have the budget and the time to do these interviews I think that's the most effective. Great. Great question Micky. Thank you. Other questions in the room? Yes. Hi thank you for your presentation. I'm interested to know if you find any distinction in best practices around this when it comes to the medium of communication like if there's a tendency to lean in towards complexity or simplicity if you're delivering something over television versus over radio or versus over print and how that complicates messaging? I must admit that I have not studied that and I'm not even sure that I know about studies that I've looked at that but just from personal experience it seems to be that written reports tend to be more complex but there's even a tendency for experts to use complex language in like text messaging and warnings that go out when it's really not the right place for it. Excellent and do we have any questions online? No. I was going to ask one follow-up question Wendy before we switch to the next panel and when you're facing a situation as you are in many maybe not in warning situations per se but in many other communications contexts around tropical cyclones for example or climate change where you have a mixed audience and there are you know that there are members of the expert audience that feel like they need to use technical terms to be precise how do you deal with the tension between the two in those kinds of mixed audiences? Well even for mixed audiences it's generally the case that people who are good at reading they're highly educated the simple messages tend to be more clear it's only when you talk when you design communications for your own expert community that it may be important to use that complex language but if you must use the complex words then I would say sure use them but then also describe what you mean by them in everyday language so you can target both audiences in your messaging but I'm not sure that it's always necessary to use those complex terms. This reminds me of David Badescu's advice on communicating where you're better off using the numbers and the words instead of skipping one or the other so thank you very much Wendy you really appreciated this. Thank you. So now I have the pleasure of moving to our panelists we have Lace Padilla from Northeastern University and Jessica Holman from Northwestern University we didn't plan it that way and Lace are you there? Yes yes I'm here I will turn it over to you. Okay great thank you so much for that introduction my name is Lace Padilla I'm an assistant professor of computer science and psychology at Northeastern University and I also served as a disaster risk management and behavioral consultant for the World Bank now I have studied hurricane visualizations for many years and I'm going to show you some of the work in that space and then focus in on a specific study to talk about in detail in some of this early work we were looking at different ways to represent the trajectory of the storm including the cone of uncertainty down in the bottom left and a few other techniques that we came up with we've also looked at animated techniques to represent the size of the storm and its uncertainty and also the wind speed estimates and their positional uncertainty with animations we've further looked at ways to represent the size of the storm and the distribution of uncertainty around the potential size of the storm we have further work looking at how people do different tasks and think about both size and intensity when thinking about these ensemble displays and the cone of uncertainty and we have more recent work where we've come up with new techniques to represent the trajectory the intensity of the storm and the size of the storm more recently we've been looking at different biases that people have when understanding these now I want to focus on the type of visualization you most often see which is the cone of uncertainty it's based on a pseudonormal distribution with a 66 confidence interval now we as scientists wanted to understand how people reason with these and what we did in some of our earlier work is we would show people these hurricane forecasts and ask them to estimate something that could help us infer how they interpret these visualizations in this case we asked people to estimate damage and we would show people location on the map tell them that this was an offshore oil rig and have them estimate the damage that would incur at that particular location the reason we do this is that we don't like asking people about probability estimates the true probability of the hurricane affecting any one area is actually zero or any specific location is zero and people are very hard have difficulty interpreting probability so so that's why we chose to use this damage rating scale and damage is nice because it is complex it includes uncertainty and it can some way help us get indirect evidence about how people reason with these now we've compared the cone of uncertainty to an ensemble technique which is on the right that my collaborators came up with the way that they generate these is we take the same underlying forecast model used to create the cone of uncertainty make small perturbations to the model and then sample from perturbation space and each one of those lines represents one of those samples i'm recognizing i'm using lots of jargon which is you know not what the last speaker said to do but that's technically what what these are now the nice thing is that they show the distribution of the possible trajectories of the storm very intuitively if you don't know all of that jargon you can start to see where many of the lines are grouped together and areas that are less likely for the storm to go now we compare the cone of uncertainty to that ensemble and we do so at two different time points in the forecast either two days along the forecast or four days so i'm going to move this up i'm going to focus on that top time point for the cone of uncertainty bring those points down and plot it on our damage rating scale one is low damage seven is severe damage so here are the responses from our participants and first thing i'll point out is that we see a steep drop-off in damage that just happens to correspond with points being inside and outside of the cone so essentially people see the cone and they think there is an area of damage and just outside the cone there is significantly less damage and this is uh there's an underlying distribution there so that's a misconception we do the same thing for ensemble we bring the points down plotted against our damage rating scale here are the responses and this looks a little bit like a normal distribution with three standard deviations we're starting to see a kind of a nice steady drop-off and there's no inside outside the cone effect so that's at least one misconception that people don't have with our ensemble we can do the same thing for the other time point we'll bring those dots down and here are the responses point out to you that overall people have a very similar pattern but they think there's going to be less damage at two days in the forecast than four days now that's certainly one of they think less damage which is another misconception we see a similar effect for this inside outside the cone where there's a steep drop-off in damage happens to correspond with points being inside and outside of the cone all right so we'll move over to our ensemble plot that and this looks a little bit like a normal distribution with one standard deviation so in some ways what we're seeing is that people appreciate that we are more certain about where the storm will be in two days versus four days i'd like to point out that this is very complicated probabilistic data that varies over time and space and people are starting to intuit it and we're getting evidence of that based on their responses all right so basically what we find is that when people see the cone they see a danger zone and when people see a visualization like on the right they understand that there's a distribution that brings us to our first major concept which is that intervals create conceptual categories when you have distributional or continuous data and you visualize it with an interval people no longer see it as continuous or distributional they think of it as a region or a category you might ask why is that the case barbitroverski writes it nicely and saying framing a picture is a way of saying that what is inside the picture has a different status than what is outside of the picture you're someone from the general public or really all of us the way that you experience things in the world is if you see a boundary or a delineation it is intended to tell you something important the picture is here the wall is here here's the fence etc so you have experienced the meaning of these delineations but when we look at our hurricane cone of uncertainty here this is a 66 confidence interval so is that a meaningful delineation well if you live here where that little dot is placed you might think this delineation is meaningful therefore i am safe and areas just inside are in danger but if we change that delineation to be like a 95 confidence interval all of a sudden you would be encompassed by the interval so the location of the exact boundary is set by the visualization designer so it's not inherently as meaningful as other types of delineations we experience so bringing to us to our second major concept which has to do with these different rules or conventions we've just talked about in the world when we see a boundary the rules or conventions are that it conveys important delineations when we see maps there are additional conventions that are associated with those a map like this for example there is a correspondence between pixel width and some type of distance in the real world for example you might have a scale and you can literally measure areas on the map and do a calculation about physical distance in the world so there is a direct mapping between physical width on a map and physical width in the world if we look at something like the cone of uncertainty it starts breaking all of these conventions the boundaries are set by the designers so they're not necessarily hard delineations and pixel distance of that cone does not mean distance on a map it means increase in certainty so we are having to suppress a lifetime of mafios and reimagine physical width as something abstract like uncertainty and that is very very challenging our second point is that conventions cause convention misalignments cause errors and before I move on I just want to point out that when we're looking at these visualization techniques and we've tested a wide range of them one of the types of conventions that really plays a big role in this is when you're showing these cones growing in size because they're literally taking up more room people think that the storm is growing in size and that also holds for this version of the cone that has blurry boundaries when we first conducted this work we really thought that the blurry boundary version would be the ideal visualization technique but we later found that in fact leads people to think that the storm is growing in size and we've also done studies where we've trained people to understand that these do not show the storm growing in size but they still make their judgments as if that were the case so it's a very very hard to override the other major thing that I want to point out and why it's important to do testing with these visualizations is that of course we found this this boundary effect with the cone and we didn't think we would for this blurry cone on the side but I'd like you to look closely at the screen and see if you can actually see banding happening here this is a perfect gradient but on different displays you start to see this banding happening in the gradient and it doesn't look continuous I drew it in here based on what it looks like on my monitor and so when people are actually using this they treat it like the cone of uncertainty with these kind of bands or boundaries not as a smooth gradient so they still have this categorical perception and that happens differently across different displays based on on your resolution so forth okay our most up-to-date version of communication is this ensemble visualization on the right and part of what we're trying to do here is try to disentangle these properties that people assume are in the visualization but what we can show on the version on the right is the path of the storm with the lines and we can also also show the category of the storm and uncertainty in that with the color and we can draw a circle on top to show the size of the storm by doing this we're trying to be very clear about the different elements that people tend to combine in their mind when they're seeing something like the cone of uncertainty just to summarize we've learned today that intervals create conceptual categories and also that invention misalignments cause errors I'd like to thank all my many collaborators and funders and here's some resources if you want to learn more about these techniques thank you thank you so much lice and we'll move to questions after Jessica's presentation push a button in front of you yeah oh there we go okay um so yeah thanks for having me I think I'm gonna maybe further problematize or describe some more challenges um building on what lace has gone through I am the genie rometti associate professor of computer science at northwestern university um and so yeah I think at this point in the workshop it's probably clear that simply communicating probabilistic information is not enough it really matters how you do it and we see several different types of sort of broad challenges that we can talk about as both bias and variance so on the one hand many expressions like text descriptions like highly likely it's very probable lead to high variance across people in interpretations and we also see you know problems where representations like the cone of uncertainty lead to certain biases like these categorical errors lace described which we could also call deterministic construal errors where people see probability and they try to turn it into something deterministic like storm size and so I think before I talk about other techniques that work it's worth just stopping and noting that the reason that uncertainty visualization and communication are hard is not just a problem of like a lack of good ways to express it it's really a problem with probability so probability depending on your statistical philosophy is either a hypothetical long range frequency or a degree of belief conditioned on certain information and in both cases people really struggle because any event that we describe with probability is either going to realize or it's not and so we're asking people to think about something very abstract and hypothetical and so many of the strategies that work are about making probability information concrete in some way back in the 1980s a cognitive psychologist Gerd Geiger-Enzer started to do work on framing probability as frequency so rather than saying something like 30 percent let's say three out of ten times this event will occur and he found that when people were asked to do classic Bayesian reasoning tasks like reasoning about the probability that one actually had a disease given a positive test result they tended to do better just with the simple trick of giving them the frequency instead a lot of my work has looked at how do we take typical visual representations of distribution things like these density plots on the bottom left here or other types of statistical constructs like confidence intervals that we might express with error bars and provide a visual frequency framing so on the right here we're just seeing a frequency based representation of probability distribution function we call a quantile dot plot and the idea is that when you give people this metaphor that probability is actually just the frequency a number of times out of some base number of times that we expect something to happen we see better judgments and decisions under uncertainty and experiments better Bayesian reasoning and so there's benefits of making it concrete through frequency as lace already discussed these sort of ensemble displays that she was presenting are one way to make the the probability more concrete here in the tracks and so you know this works in a hurricane context as well but one of the themes of my talk today is going to be that even when we try very hard to make things concrete we can still see these various errors and so in some of Lacy's work that she didn't quite get a chance to discuss they asked people to provide damage ratings at several locations given these ensemble visualizations and so the two locations A and B people had to provide a damage rating on the left 98 percent of the people in this study said that location B would get more damage than A and that is correct B is closer to the the storm center or the the mass of the distribution however 36 percent of people said on the right that A would get more damage and that is not correct so again B here is closer to the center but you'll notice if you look carefully A is actually intersecting one of these little ensemble lines and so people even though you know they've done a lot to get over this you know size uncertainty as size you know misinterpretation people are still doing what we would call this deterministic control error they're taking a representation of probability and turning it into something that is concrete in this case you know whether you fall on one of these lines is thought to matter whoops and so you know the types of categorical errors are deterministic control error that lace discussed is where we're interpreting you know probability as some attribute usually of the storm its location its size another type of very common kind of heuristic or shortcut that i've seen all the time in my many years of research on uncertainty visualization is something the economist chuck manskey calls as if optimization and the idea is that you know people want to use a point estimate something like an average in order to make some decision under uncertainty that we really shouldn't make using a single point there's this famous quote attributed to Lyndon B Johnson where when he was shown an interval forecast by an economist on his team he said ranges are for cattle give me a number and this is extremely pervasive and often people don't know that they are implicitly suppressing uncertainty in favor of a point estimate and that's what makes this so challenging to design for so in a hurricane context you know whenever we're giving a probability we should expect people to try to turn it into something certain so you know in a case like where we're trying to give someone the probability that their home might flood say it's 22 percent in my county you know no one knows not no one but most people don't know what to do with 22 percent again this is a hypothetical quantity and so you know it's very common in this kind of setting for someone to think well this is pretty far below 50 percent therefore I'm just going to assume I'm good and so we see you know rounding as one expression of as if optimization we're making something a concrete single number that we can deal with in a visualization context a corollary of this is that the extent to which a visualization of distribution makes it possible to just focus on a single point estimate like the mean usually predicts how much people are going to suppress uncertainty and make you know non-optimal judgments or decisions so you can ignore the um sorry the annotation that should not be there but here we just have means with confidence intervals you know this is a very typical representation if we just have you know 2d uh distributional information and it's very easy to just ignore the uncertainty because it's a separate mark we can just look at the mean however what's less appreciated is that many you know more expressive representations of distribution things like density plots or even these frequency-based displays it's still implicitly you know visible what the central tendency is and so what we see that people I think don't predict is that even when you're not giving someone a point estimate specifically many users will still kind of focus on the point estimate so with a normal distribution I can see where the center is the mode which is also the mean and so people will suppress uncertainty so how could we design you know displays that really make this hard to do some of my earlier work looked at probabilistic animation where we call this hypothetical outcome plots we're actually going to take draws from the joint distribution we want to display here and we're just going to visualize those as frames in an animation and so now if I were to ask you what's the mean location of the red bar what's the mean location of the blue bar it should feel harder and you know there's many ways reasons to think that this kind of animated display while it looks visually complex is actually you know something that people could use to kind of intuit you know information about probability so temporal frequency people in vision science have found is is something that people can get probability information kind of implicitly from it's sort of visceral and becomes intuitive in a way that often static representations are not we can also apply this to any type of visualization as long as we're not already using motion or animation so here these are rainfall plots and I'm just taking draws again from the underlying distribution here a posterior distribution of my model and animating that however again I want to problematize thing and and talk about how hard it is to really fix some of these errors what we found when we started studying you know how well do these types of plots help people suppress uncertainty was that depending on the conditions in which someone is using a display even when you try very hard to force them to actually integrate the uncertainty into a judgment it can still backfile or backfire so in one study we thought that if we added means these lines on the bottom explicitly then people would probably do much worse because suddenly they would have this point estimate and if we ask them you know what's the probability that the red bar is going to be or the red value is higher than the blue value suddenly they would they would ignore the uncertainty in that judgment what we found was actually the types of effect size judgments and decisions people made were virtually indistinguishable and this was puzzling to us because we thought we were doing everything possible to prevent them from just you know focusing on the means I should note that when people you know were asked you know what's the probability that the the red bar is greater or the red value is greater than the blue value often what they were doing when they when they see means is they're looking at the visual distance between central tendency and then just mapping that to a probability judgment and so they're they're basically using a heuristic or a shortcut like I said though we found that people were were answering very similarly even when we suppressed the point estimate and what ended up we ended up finding out by looking at how they reported using these plots was that even though you know we were intentionally trying to to force them to contend with uncertainty many people were looking at the red line jumping around which you know represented the score of one team in this in this fantasy game that we had them play looking at the location of the blue trying to estimate the mean of each of those and then comparing the visual distance between means and so basically people are doing exactly what we tried to thwart and so the the point of this even if you didn't follow all the details is just that no matter how hard you try to design something that will really you know force people to internalize uncertainty it can be incredibly hard because these heuristics these tendencies to suppress uncertainty are so pervasive we do know from some of my own research and other research that this kind of suppression of uncertainty is more likely under high cognitive load when people you know already have a lot of information they're anxious under low graphic or statistical literacy as well as under motivated reasoning which you can think of as sort of you know in situations where people want a simple answer and they want a particular type of simple answer this is where you know we have to be most concerned and really careful that we're giving displays that don't support this kind of just suppression of uncertainty so one implication of this is that you know people do come to forecast with different situations with different levels of experience and so we want to provide information at different levels of granularity to you know serve people with different needs but at the same time we want to anticipate that at every level no matter how much we aggregate the information or how detailed we give it people are going to be trying to just turn that into something more certain and so I think we can learn a lesson here from election forecast displays so in many ways when people go to an election forecast you know something like provided by these news organizations they want a simple answer they want to know you know should I be worried about this election you know do I need to really you know be concerned about voting etc and I think that's similar to when people go to a hurricane forecast they want a simple answer that's going to help them make a decision like should I evacuate and so in 2016 if you looked at 538 election forecast top level display when you first load the page they gave people two probabilities of each candidate winning and what you'll notice that it's very easy you know if you don't want to grapple with uncertainty to just round this value up and so here you know I don't know what to do is 71.4 percent but it seems pretty far above 50 percent so I might you know just assume oh yeah you know Clinton is going to win this election you know four years later I think they learned some major lessons about how likely people are to just suppress this and so the top level forecast you get no probability you get no point estimate at all you have to contend with the uncertainty you know we have to sort of look at how many you know plots are there and make a judgment a visual judgment and so this this type of approach using frequency to try to get people to internalize things is still our best bet but but of course it can like I said backfail or backfire you know some other work that laced did not get a chance to present is also relevant here so another implication of as if optimization is that even when you give people more detailed probability information they might end up using it in such a way that their judgments are very similar to if you would really aggregate it or summarize that information so here we're seeing two different depictions of uncertainty in location about you know in a hurricane on the left it's a 66 percent interval on the right we're also seeing size because they they're showing these splats which are both information about the location and the size predicted by the model the actual plots were animated so they both did show size information the one on the left would would animate over time but the point was that the judgments of damage were very similar between these two plots even though one is much more detailed often when we design displays we do have to summarize to some extent you know we can't give full you know continuous probability and and here you know in this study when they when they give these three different intervals the judgments were more different from either of the other two displays it's all about really finding the right level of aggregation and testing it with your users which I think has been mentioned already by Wendy so you know in this display in the middle you know we're still not getting uncertainty so if we ran the model with different initial conditions for instance the projected location and the and the uncertainty in the projected location would change and so here we might want to show people not just you know a single representation but show them multiple possible scenarios and so this is my final point that often you know we don't just want a single map we want to show people you know several different scenarios with a narrative that's in simple language that can help them understand like these are all things that can happen and so you know one kind of you know approach I like is just this simple strategy of let's show people a couple you know representations visualizations of scenarios that would not surprise us so things that we would expect to see a lot that are from sort of the center of our you know posterior distribution from our model then we can show some things that we you know might see sometimes or that would surprise us a little as well as things that we are we don't think are very likely but if they happen would have detrimental sort of consequences so and we can play with things like visual emphasis so how big do you make these different scenarios in terms of you know each scenario is kind of the visualization how much visual weight do you give them can really affect how much weight or how much you know probability people then assign to that so again it takes experimentation and it takes testing with the people that are going to actually be using this so yeah I will stop there and happy to take questions. Great thank you Jessica. Sure so we've run a little over but we have two things that we wanted to do it to conclude this section first is to introduce a prototype and have each of our our speakers talk about the prototype a little so we're going to cut that a little short because we've run over and because we'd like to save time for at least a couple of questions and Andrew would you mind moderating the questions after the prototype Great thank you very much. Okay so we've heard a lot about risk communication especially graphical risk communication from our experts and a lot of this is based on theory and a lot of experimentation I in my former life was a person who made some of some of these graphics that go out I will take full blame for this so what we thought might be an instructive few minutes here would be to take our experts and ask them to be take some of this theory they've just discussed and apply it to a very very young very very naively built product that's supposed to express the uncertainty of receiving certain hurricane force winds so I don't know if we have the am I sharing the graphic I'm sorry oh I had sent it okay I thought it was in the stack too so um so while Andrew is pulling that up do we have any burning questions from the audience yes yes I'm surprisingly easy hi everyone uh kassandra shivers Williams I'm from NOAA's a weather program office and I'm wondering if either of you have insight into how we might be able to either combat the heuristics people use when they're looking at these displays and trying to make decisions or maybe redirect them or re-emphasize them in a certain way right like maybe their baseline needs to be re-anchored somehow or like how can we help people I guess interpret differently because we know that people use heuristics with decision-making especially like you talked about with high cognitive load and things like that we can't really fix that in some way so wondering if you guys have thoughts on that yeah I guess I can start um I mean I think it's hard I was trying to sort of make clear that you know it's never it's never easy um but I do think the frequency framing works well and so you know rather than just giving what someone a single graphical representation that includes the probability show them multiple scenarios or multiple you know draws from that same probability distribution separately um and so you can do that using animation and stitch them together or you can just show static what we would call small multiples which in visualization which is like a set of static stills that show you know slightly different information and so each one corresponds to you know some point in the distribution you're trying to convey um I think that's you know still going to be one of your best bets because people then don't have to interpret things like these visual cues like how fuzzy something looks on the map or the size of it as probability they can just see that like oh this happens and this can happen and this can happen and this can happen and that makes it much more concrete so I um you know I still think that's one of the best approaches and yeah as lace discussed you can apply this in a single display as well it doesn't have to be separate little maps um like her ensemble plot um lace yeah I agree with that and I think that I think expecting people to do anything different or be trained or have you know more exposure to something is just generally a technique that can is just brought with failure in my experience even when you try to train people um what happens is the visual information guides their decision so if you want people to um in some ways respond differently you have to change the visual information rather than added an annotation for example and I think just got really good examples of ways to potentially change the visual information to help people come to a better conclusion thank you both and I just like to add that Sarah Coot has done some some research showing that the type of information that the way you present the information may be different for different kinds of decisions and may be more effective depending on the type of decision they're facing so it's there's not a one size fits fits all answer for this so thank you all very much and I'll move it over to Andrea all right so sorry for the technical delay um so for this exercise I'm going to show this prototype product um just to note this is not an official hurricane product this is not even close at this point this is a product that has been paid for through research and development um created by atmospheric scientists and the point here is that maybe this isn't the time period when we would expect to necessarily start bringing in social science ideas but a lot of these products some of these initial design decisions sometimes get momentum and we get to the point where we create this great product it takes us years of development we get it into operations which means putting it on machines and then we do the social science research and it's seen that the social science is holding us back from releasing this great product or we've released it and then we have to change it later so the whole idea here is maybe it's time to start looking at these earlier products and and you know maybe it's time for these different academic communities to actually start understanding each other so we can make better design choices sooner and not create this propagating issue so here are two versions of a graphic that is meant to express the maximum wind speed exceedance values for the next five days I know that the title itself is probably something we could talk about for 20 minutes um there are two graphics here one represents the most likely wind speeds uh that people will see and the second graphic represents what we call a reasonable worst case scenario another term I think we could probably talk about for 20 minutes um and we asked um our our speakers to please give us some impressions what would theory tell us that we really need to work on here right away how can we improve this product now before we send it along so have at it I mean lace do you want to start sure I think something to to think about for the general public is they will focus on the visuals and they will ignore or maybe not even be able to understand the text um so if you have a situation where them thoroughly understanding that the textual annotations that that will be very challenging so in my interpretation of this I think people will look at the visuals and very similar to the cone of uncertainty that doesn't show any you know intensity or size it just shows the track they see it as impact area so I think people look at this and see impact area even though and maybe that is not that wrong for this given that you're trying to show wind speed but I think that would be the misconception I would think that they might come to for this um and then of course going to the categorical perception where they're they're going to look at the areas that are in the brightest red and say okay well if I'm not there then maybe I'm okay um that's kind of my first impulse of what of what might happen one thing I well know is this is actually impact area so but it's very understand how this could be misconstrued due to the fact of the cone of uncertainty is so ingrained in people's minds they may not actually see this as what it is right so so thinking about that you know you're you're communicating um wind for speeds right and that is kind of hard to map onto someone's personal experience one way to communicate the same information but maybe in more plain language would be what is the impact to them maybe it's that areas that are in the light yellow are at low risk or maybe infrastructures that are unstable might be dangerous but areas under the red um you know major hurricane force winds any type of structure might be at risk so instead of having the technical terms here and harkening back to windy stock it could be useful to remap the the categories to people's individual experiences so they have an understanding of of what their response should be in relationship to this great I would add um you know you have like two different scenarios say and you know people will probably interpret the bounds on those you know um different wind categories uh they'll put place too much weight on exactly these two scenarios and so you want them to know that there's some uncertainty you know are you in the hurricane force wind or not is you know like it's not exactly this line and so I would probably use something like I suggested at the end of my talk where you show people a few more scenarios that are highly likely so I think we have on the left like the maximum likelihood estimate which itself only has a little bit of you know probability assigned to it so I would probably draw from like the center like 50 percent uh of the of the density of the posterior distribution of your model and um and say you know narrated as you know these are these are all different scenarios that would not surprise us something like that and then you know you could still have the worst case um you like right now I think you're sort of implying people should put equal weight on these two scenarios when in reality things like the one on the left are going to be way more likely if I mean maybe that's because you know people will underweight the very risky but low probability event so anyway depending on what you know how much weight you want them to give each you could you know have multiples you know examples that are similar on the left of like things that we think are likely and then have a much bigger sized one of the detrimental like a that's where I would sort of play with the effect you want to have well excellent um we're fortunately out of time but this is something that I'm sure we could have spent a lot of time and I think this really emphasizes how co-development is so important here um just letting atmospheric scientists develop one thing and then take it over to social scientists for feedback after we've kind of already set our minds up on certain things is not the best way to go about so I really appreciate your expertise and just I know you could immediately find something wrong there so no I think it's a good start thank you thank you lice okay we are at the last panel of speakers for the workshop and um one of our speakers unfortunately cannot be here because he is briefing the california governor about what's happening in california however we do have two excellent speakers who will be presenting to us today about access and functional needs related issues and communicating risk we're going to start with Joseph Trujillo Falcone who will be speaking to us about Spanish related risk communication issues and possibly other things and this will be followed by Sherman Gillins who is with FEMA's office of OIDC I apologize I cannot remember exactly what that ODIC office of disability integration oh all right thank you very much and um you do have a little bit more time because we have one speaker who's not here oh wonderful thank you so much and good afternoon everybody it's such an honor to be part of such an amazing group of panelists and speakers and honestly I'm very humbled to be able to uplift the perspectives of two groups that I believe aren't often talked about enough when it comes to tropical cyclones and or those are multilingual populations and immigrant populations the national weather service has linked fatalities to language inequities all the way back to 1970 nearly a half a century later we have yet to develop a thorough way of communicated emergencies in other languages the fact that we continue to have a monolingual emergency communication system does limit a lot of individuals from really understanding and engaging in the information to make proper decisions a lot of my work has actually been inspired by these communities themselves before I ever conducted any sort of social science research in this area my first job was a bilingual meteorologist for a radio station in the local brian and college station texas area and literally I know that my first major weather event would be two weeks into my first my career and it would be hurricane harvey it was an eye-opening moment I was the only Spanish speaker in that area that was able to provide that given information I got to witness the first hand accounts of individuals and really got to see these language vulnerabilities first hand I wanted to kick off today's presentation with more of a personal narrative and tie in some social science research really illustrate the disparities that exist and continue to happen present day I want to start off by a comment made by a viewer manuela cruz in the aftermath of hurricane harvey her among the entire brian college station texas community were extremely confused about this event different media outlets were communicating different things in a language that she was not familiar with and she expressed that if it was not for this information in spanish she would have not been able to engage in it and eventually to take take protective action I can name a ton of tropical cyclones where this continues to happen hurricane Ian hurricane harvey hurricane Irma the list goes on and on and on you continue to see new cycles where language inequities contribute to people not understanding and engaging in information and present day we're trying to continue to increase those efforts and reach these communities but it was of utmost important during this event where I got to personally experience it and see it for myself that when you do provide information in someone's dominant language they are able to engage in that information but here's the thing when it comes to even the most principle and even the most foundational definitions in risk communication language we come to see that you can look at whatever agency whether it be the national weather service or the world meteorological organization the fact that as of right now we cannot yet agree on a defined definition for hurricane watch and warning tells you that entire story whenever we go about translating something from english to spanish messages can get lost in translation and the lack of an official english to spanish dictionary here in the united states for weather climate and risk terminology inhibits us from creating that consistent terminology going forward i've published a lot of this under this area within the bulletin of the american meteorological society with tornado hazards but it really does apply to hurricane hazards as well whenever we go about translating something from english to spanish we you know there are different variations regional varieties dialects of our language that could really get in the way of understanding and so say i'm from beru i can talk a little bit differently as somebody from puerto rico or para guay does and that is very very important our language is beautiful and diverse and it should stay that way but when it comes to emergency messaging when we translate something from english to spanish we may do it in our own best capacity which may not resonate with everyone in that given community one of my favorite examples is actually the word rip current because in spanish the translation of it for the part of weekends is the word resaca and so the weather show is issuing avicios de resacas rip current warnings for the coast but yeah that may mean rip current for a few people but for the rest of latin america a resaca actually means hangover so they were quite literally issuing hangover warnings for the coast which technically correct but i mean wrong hazard but that's the fact of the issue here the fact that we can't consistently communicate in someone's language and aren't able to portray that risk messages get lost in translation all the time and we have to make sure that we go about doing it and so whenever we go about approaching this sort of work and really and when someone comes up to me about okay well how important could this really be i want to be able to actually give you a big number here 69 million individuals speak a language other than english here at home in the united states as of the most recent census bureau data that is one in five americans already present day by 2060 there are reports out there that estimate that nearly one in four americans will see speak speak spanish at home for example this number continues to grow in size and proportion and it's not a matter of if but when this becomes more consequential in our emergency communication systems going on into the future and so topic number one of course is that language inaccessibility i highlighted a lot of examples here in spanish but it could be applicable to all the other non-english languages spoken here in the united states the second theme that i really wanted to emphasize a little bit upon and something that we don't often bring into the conversations of tropical cyclones is the conversation of immigrant populations and that this particularly undocumented communities a us customs and enforcement have this thing called the 100 mile border zone where their jurisdiction lies 100 miles between any sort of land or sea border and that involves nearly two of three people here in the united states they're able to open up for example checkpoints along the border so that they can make sure that people are us citizens they can also be able to search people without a warrant in these given areas that are highlighted in red there of course you could see how this can become an issue when it comes to hurricane evacuations when people are trying to leave the coast there are undocumented immigrants within these communities that have a choice to make do you evacuate during this hurricane or run the risk of never seeing your children or your other family members ever again let's take a moment to think about that no matter the risk messaging no matter the condition the fact that you might not be able to see your family again it's something that really really traumatizes these individuals and it may inhibit them from taking protective action altogether i got to experience this first hand as a broadcast meteorologist during hurricane harvey when the president of the united states at that time was saying that immigration customs and enforcement would remain open during evacuation orders while local officials were communicating and saying that no no no no we're gonna actually remain closed and make sure that everybody stays safe this actually made the texas tribune and lorella here from aclu was just stating of how inconsistent the messaging was during this event and it actually caused a lot of immigrant populations to stay back and not evacuate during this given event it was a very eye-opening event and you can look back at other hurricane hazards for example in catrino where you know people from immigration customs and enforcement would go in to try to help the community but would end up setting for example in catrino three people to deportation proceedings in the aftermath of the rescue efforts i'm a fond belief with lorella here that everyone no matter the color of their skin or background is worth saving during these events and in the most recent administration there has been efforts to be able to lift this temporarily for the time being you know these checkpoints will not be open during times of evacuation so that everyone can take this decision decision altogether there's still a lot of work here to do but i am a fond believer that everybody deserves a chance of safety when it comes to hurricanes and any other tropical cyclone events out there moving just beyond the language and the immigration standpoint of all this is the fact that we also have to make sure that languages or messages are tailored in such a way that it resonates with the giving community you know like i mentioned before being peruvian i've never experienced a hurricane before but someone in puerto rico has lived it all their lives the fact that we categorize big broad groups like hispanic or latinos is one singular category can really get muddily in the way we tailor messages all together because if you try to teach someone from the caribbean what a hurricane is they might feel offended at the fact that you're trying to do that whereas there may be other hispanic or latino communities that may need it more because they've never experienced that hazard all together so i always like to tell my colleagues that sure a lot of this language research that we're conducting is a wonderful first step forward but it should only be considered a first step forward you can translate the message but without the context and without the cultural variables as well you're not able to properly contextualize it and tailor it to a given community i wanted to end today's talk with some things that we're working within the nation oceanic and atmospheric administration i'm actually a principal investigator on a grant that supports this initiative and we are proud to actually be able to integrate artificial intelligence and being and moving forward and creating language translations for not just messages in spanish but vietnamese mandarin among other languages as well what we're trying to do is create a neural adaptive translation software with using artificial intelligence to where they have access to a national weather service domain and of course language translator certified language translators that understand the different dialects of the language are brought in we're able to establish glossaries and definitive terminologies and as of right now it has about a 98 to 99% accuracy rate after years of training and we actually were able to use it for the first time during hurricane ilari to be able to translate the key messages completely into spanish using this software the big thing within the national weather service is a lot of the translations conducted within the national hurricane center and other areas as well are done by the san juan office and done by voluntary translation teams within that agency as a forecaster you have the ability to not just be able to forecast but now you also have a lot of responsibilities in translation it's a lot of work and we're trying to reduce that that workload for these individuals so that they can at least have some automated translations they can just review a couple of terms but we found a great success in this area to not only communicate key messages but also be able to translate information into other languages like for here vietnamese and also contextualize the cultural component as well so that we're able to bring people from step one and say this is what a hurricane hazard is this is what you need to do something these are the recommendations that are involved in this and let's break it down from the very first step and go all the way going forward as a p i on this grant we're actually creating software within gis to be able to locate where multilingual communities are in the united states so that we can find these populations tailor it find community organizations and be able to work with them in the future i invite you to scan this qr code it it's actually a glimpse as to what the weather service can and will look like in the future weather dot gov slash translate you'll be able to get a first glimpse at some of the tools that we have been working on to be able to provide language accessibility because ladies and gentlemen it's been nearly half a century we have not made tailored solutions and we'll continue to work to make sure that we can create a weather ready nation for all thank you so much ready to go backwards to my slides here can we tee up the slides am i controlling them with this i think that he was going to get it going in the back while he's doing that i'll just uh first of all say hello everybody um around this time last year i was uh i think i was in Selma Alabama after a i think an ef2 or three tornado had just touched down and reached havoc on on one of the communities uh in the Selma area and i finished the year 2022 in Clarksville Tennessee where another set up tornadoes touched down and reached havoc and that in those communities uh turned out to be a pretty busy year for us i think it was probably a record year for billion dollar disasters uh 2023 and if there's one thing i take away every time i go to these places is that um you learn the most by talking to people especially when you're talking about communication and what resonated by finding out what didn't work um i find that in most cases it's not the message it's really about this way that people try to see circumstances in a light most favorable to them and their circumstances and how they process it so i'm going to talk a little bit about that through the lens of disability integration um because while we know that there are people with disabilities in society uh it's been my experience that they're often invisibilized during disasters because they either don't appear they can't get to places like shelters and disaster recovery centers um and a lot of times they weren't considered in the planning and i think that becomes most acute in the uh the way we communicate with the public whatever we uh uh warn them about what's to come so we'll go to the next slide i became FEMA's fourth disability coordinator in august of 2022 um just the the the law that established it was the post-catrina emergency management reform act and i bring that up because uh there's a bit of serendipity at play where this law in my experience is concerned what's not apparent there is i was i'm in a wheelchair right now in case you didn't see me come in um i was injured while serving in the marine corps and i spent a great deal of time in the san diego area as a person who had just joined the 61 million club the 61 million people in the united states who either identify or are regarded as disabled um that's important because i processed what was before my injury a little disaster experience i really didn't know i had or didn't realize i had when i was in the marine corps we responded to hurricane floyd evacuated peris island and did some work around that so i had a little bit of experience i happened to have the platoon of all the recruits who were injured uh uh medical rehab platoon so i had to move about 112 uh uh recruits about you know 100 miles south into into georgia and that was an experience um i also was in japan when the kobe earthquake happened this is where the beef kobe beef comes from but there was a big earthquake there and uh and i also did some work with harvey maria before i even thought about coming to fema um and the point is disasters touch a lot of lives and waste people don't realize and a lot of times they try to forget about it um and what that means is they don't learn from it they don't learn from what happened and a lot of times if they're lucky they kind of see that as their reality going forward and that's unfortunate because um and as i'll talk about later unfortunately people do die um even after having experienced disasters because they make the wrong decision um and i said as i said before you know this this whole idea of seeing the disaster or the circumstance in a light most favorable two one circumstances it's not theoretical for me um next slide please i heard the uh previous speaker talk about um uncertainty suppression i took i took note of that miss holman because uh that's an interesting way to uh sum up what happens in these moments and i'll talk about this picture which is from hurricane katrina and what that had to do with my experience almost 20 years later hurricane katrina happened in um august of 2005 and like most of you probably we watched images of people at the at the football stadium um i don't know that they focused so much on disability but uh but the reality was if you were disabled or on an advanced stage you were more likely to die in hurricane katrina than most other survivor populations now as a person who was newly disabled this was in 2005 i just got now the marine corps three years prior to that i saw this and um i wasn't attached to it but i did note that wow if i was there i'd be in pretty bad shape and again this was in august of 2005 um anybody from san diego you remember what happened around labor day weekend in 2005 we'll click the click the next one another image is going to come up there were wildfires this was just a month after i'd seen all these people sitting outside the stadium saying to myself boy i'd be in a bad situation if that were me so when i woke up on that i think it was monday morning and heard about wildfires that were up in the la area i thought wow that's not good that's that sounds pretty bad for those families out there you know i wonder what they're going to do i can't imagine losing everything i went about my day a little bit and um went back to the television and i think i saw maybe a crawl they talked about the fact that the fires were moving closer and i lived in rentropenis kittus at the time and if you know anything about wildfires they they moved pretty fast there's there's not a lot of warning but in this case uh i could i could see the sky getting red and i could taste the soot in the air and by the time you know the warnings were coming across the crawl on the television saying um this is a time to evacuate if you're in the area i was in uh qualcomm stadium which is where the san diego chargers play was a site that they advised us to go to and the first thing i thought about was that picture i'm not going there i am not going to put myself no now mind you there's a fire bearing down and in that moment that that sort of that that window of uncertainty suppression i'm making a calculation now i'm a fairly educated guy i graduate from the university of san diego i think i'm pretty smart but in that moment it became less about what was most safe and more about what was most certain for me i wasn't safe either way i felt but what's more certain is that if i stay in my home and nothing happens i'm better off than if i go to that football stadium and they don't let me leave and i can't come back and even if i lose everything i'll get out in time how fast can a fire come um and and it wasn't until uh it was basically it had jumped the freeway i-15 and uh if you know how san diego set up the miramar base there's a lot of open land and it just gained steam and began to burn everywhere and kind of surrounded the neighborhood if you can see that picture and i thought in that moment i made a bad decision uh however i'm still not in that stadium where after having been to a couple of games i know it would have been worse because it wasn't about whether i'd be burned alive it was about whether they would understand my needs i use catheters to answer the call of nature they're not gonna understand that i can't lay on a cot because i'll get the cubitus also i'm rationalizing in my mind through the window of uncertainty that i'm in what's going to be more certain for my safety um now ultimately they put the fire out uh fortunately the area where i lived in didn't incur damage but i bring that experience into this role as a disability coordinator and think about why do people make those decisions part of it is well i understand it i i you know i get it but in that situation i have to say i didn't have a lot of time to think the work we're talking about here is when you when people have time to think when you see that that storm system moving what's going through the individual's mind lives with the disability whose arm is his or her sanctuary or their sanctuary and you're now asking them to leave that sanctuary and go into an instance where the uncertainty is profound to a point where it could have permanent implications next slide please i think i heard um dr Sutton that i can't remember what the event was and i and she talked about messaging and the importance of it and until i saw this graphic and this is sort of an adaptation of a graphic that dr denis milletti you probably all know who he is um got arrested so he talked about this sort of cycle or the phases of how people process and i thought it was so interesting because i can sit here and think through every part of this that was uh instrumental in my decision-making process starting with what i was being told i was being told to leave my home to go to the football stadium okay i know there's a fire but how does that you know i can't personalize that because now i'm thinking about the things that are personal to me and they're not giving me enough information what if i get in my vein and run out of gas i mean the freeway is packed i'm not going to make it nobody's going to stop to help me there's probably no more gas i'm thinking of the you know the worst-case scenario but those five elements of of the message um and i'll talk about in a little bit how that going forward informed the advocacy work i put in now but not going through those steps not telling me who's telling me this right i'm i'm listening to newscasters talk about it i'm seeing a crawl but i don't know who this is from and it just didn't seem credible right the hazard itself i knew what it was it was wildfire but i didn't know where it was i didn't know how fast it was moving all i heard was get out right and so a lot of this is being processed again in a light most favorable to the individual who then has to decide am i going to put myself in a worse situation or at least that's the impression the next part of it is how i got the message was a clear complete consistent certain or accurate um i can't tell you enough the you know when i was asked to come to this to speak at this i thought the most people even know what a cyclone is right or what a what a a um the cyclone and the uh what's the other thing the typhoon right what the differences are a hurricane right i'd been to australia so i kind of knew what a cyclone was and i've lived in japan but a lot of people can't process that and i think most people take for granted that a lot of people are going to process what they're being told in a lens that um is is mindful of the actual danger when they when they have to probably go to google to figure out what is this that we're even talking about what is a what is a cyclone i remember california i think it was last year they had a cyclone they had a bomb cyclone what is that you know what i don't you know so we talk about it in this language of uh you know of of being really specific and wanting to be clear um but clear is in the eye of the beholder so i would just that's where inclusion on the front end inclusion at the table during planning is really important because they're going to tell you i don't know what that is and even at fema i have to do a lot of work inside of those walls having people understand you're not dealing with people who do this work they may have never been in a disaster before don't take for granted how unclear you might be if you're not having them help you with the messaging so that was a an object lesson there um but then the most important part is that third part how is it being perceived do i understand it do i believe it it can i personalize it and how does that inform the decision i'll make and this is a cycle because as you all know news changes fast during hurricane in um you know we get about eight days where we can kind of see it and i'm in my office with my team trying to guess it's not a game but we're trying to game out what if it turns this way what if it does this because what we do is we look at population data disability population data we look at the census we look at um uh svi index we look at empowered data this these are all the people that are electricity dependent and we look at i-e registrations from previous disasters to make a we triangulate ourselves into a profile okay we know this area has a lot of people who are deaf right salma has a deaf school right there's i mean we look at all that information and then we try to sort of assess or diagnose what's likely to happen if it if the wind or weather system does certain things um and that hopefully guides the messaging because then we know if you've got people who are you know who are going to need american sign language and i see the governor doing a public address with no asl interpreter we're in trouble somebody better let the state know you need to have a sign language interpreter you've got a lot of people you are not going to reach people who are deaf live alone they live independently they function very well you have entire communities in Puerto Rico that don't interact with society because they have their own language and it works for them but when a disaster happens and there's information that's critical if you don't have relationships with those communities you would better be doing the work of figuring out where the gaps are so that people aren't getting missed um as you're putting out the message and then the last part is very interesting um this idea of either moving or milling and milling is when you're kind of you you processed everything and i'll use sort of the analogy because i was just in Clarksville after a tornado where you've got your family and you're trying to decide do we leave or do we go and as dr milletti pointed out in one of his videos um you know it's it's been this phenomenon where if if the dad makes the decision it's typically to stay but if you appeal to the mom and there are kids their instinct is to leave and go to a safer place um and so i found that interesting it's probably not generalizable you know there are differences but he talked about how you really have to be mindful of who we're directing information to who are the decision makers now if you're talking about people with disabilities you're probably talking about somebody who's a caregiver um whose bias may be for leaving and you may have to interact with a person with the disability and convince that person you're gonna be okay i'll be with you you know all these other aspects of living that are pretty routine uh but during a crisis it takes on a much different you you know and so we talk about the idea of milling and in that moment in those you know that 48 hours before landfall what's happening in homes who's having the discussion who's leading the discussion who's got information directed this is where cultural competency i know we talk about it at nauseam and it's sort of this virtue signaling thing it's it's a strategy if you want people to move out of certain areas you have to make them feel like the information is for them and add them right and i'll talk to you about how this became real in a very three-dimensional way for me um next slide please um was anybody in um florida after hurricane again or see any coverage on sanibel island that bridge that you see is the bridge that um the hurricane blew apart sanibel island is it's a mostly higher income area so people have autonomy they have resources the problem in this case is there was a lot of uncertainty about the actual threat and the only reason why i know this is because i talked to a woman who's blind and she was living in a trailer and the only reason why she left is because her daughter drove 200 miles to get her out of there she drove from up north 200 miles and said mom we're leaving and i said um so we were thinking about staying i mean like was that even a choice and she said well i didn't you know and i said uh well what about you know anybody who also thought that way and maybe didn't have somebody who came and she said yeah there were a few neighbors we lost one was a brother and sister who were in a home um and as it got washed away the brother was able to escape and the home and under with the sister in it and i said well you know why do you suppose they stayed um did they not get the message and she said no they heard everything they heard everything i heard i mean we sensed the danger um but they didn't personalize it we had been through this before we had seen these before right you still hear that i have friends in florida still that you tell them hey be careful they're like ah they're gonna they're gonna get soda sit out on a porch and watch the hurricane go by yeah it's just just weird um but she told me about how they heard everything everybody heard the same thing and those who could leave left those who couldn't because of inaccessible transportation unfortunately that's another aspect of this that i have to uh account for in disaster response but fema doesn't come in until it's been declared we're not there the day it happens a lot of times the disaster starts local and in locally so we have to have some sense that the community has been prepared in advance of these uh but in this case when the bridge went out um people couldn't leave and i don't think a lot of them knew that it was going to be that bad you know and so the lesson there uh had a lot to do with understanding the psychology of a would be survivor it's not always clear some of it is very cultural i've been to east palestine with the train derailment happened where about a we're on the one year anniversary um i went to Puerto Rico after hurricane fiona um and there's always this autonomy that people retain and that autonomy sometimes means at their peril and how do we break through that is the constant concern um next slide please you know and i just saw justin leave he's our communications director uh external affairs but a lot of what was different at fema uh since i got there was my deliberate decision number one to be more visible ahead of these storms and speak the language of the disability community now it does vary right it's not just one language but telling somebody to evacuate versus telling someone how to access accessible transportation so that you can evacuate it's a different way to talk about it and sometimes you know the the bandwidth is there's not enough bandwidth to squeeze all that in right we can't get every person who's deaf person who's blind person in a wheelchair person well we had better try in some way and sometimes it's not about being on fox weather it's sometimes having community having connections with stakeholders who have those channels all the time prior to the storm that can then deliver that information whether we're talking about uh centers for independent living of course the Red Cross is always a big part of it but having community organizations in in Selma it was the churches having them be a part of that and having an awareness that they're going to have to be a part of that was a was a big important aspect of how we strategize communication especially when we have these you know pre-landfall opportunities to talk through what can happen so um i think that may be the last slide is at the last one next slide please and so that's pretty much um you know the the expanse of how we got through 2022 and 2023 this year i don't know what kind of year we'll have but i anticipate that um i'd like to think that we'll be more ready and a lot of that will entail making sure that our messaging uh effective messaging means that we understand what's going to resonate uh what's going to have people overcome that that uncertainty paradox that they're caught in so that they make the best decision for themselves thank you fantastic talks thank you both um i'm inspired and i'm gonna kick off with one question and i'm gonna open it up to the audience because i know you have all probably have a lot of questions um this one is uh directed to both of you but first to joseph um are there specific words that you're learning that are difficult to translate that are jargon in particular within AI or within the disability community language sphere things that are hard to translate thank you so much for that question yeah no we've been having a lot of fun with that just looking at these one of my personal favorites in terms of jargon is actually a trough um they it often gets translated to like trough where pigs eat and so that there there's different varieties and a lot of those challenges actually tropical cyclones there's been a lot of good terminologies and especially puerto rico and the caribbean have experienced those hazards before and so they have words to describe them a lot of it has actually been a more focused on the tornado space a flash flood space and then in the winter weather space where it's especially in places in latin america they're since they don't experience a ton of winter weather hazards sometimes those words don't aren't able to even be defined like one example is we make a very heavy distinction between freezing rain sleet and snow that distinction is not as distinguishable in spanish just because those kind of hazards don't exist over there so there's no need to define those things within the tropical space though a lot of the things do relate to the risk terminology just because even the word warning as i exemplified isn't there's not a direct translation for it there's about six different variations of it and depending on where you're from you can contextualize it very differently and so we have been learning a lot through this space of looking at these sorts of varieties but really the importance of integrating linguistic experts in this conversation you know i'm a social scientist i can conduct communication research on how these people perceive these words but i'm a fond believer that you know we need to bring certified translators into the table that understand these dialectical varieties and know how to communicate it forward and so really bringing them into this conversation would be very valuable going forward yeah i think uh dr son you might have talked about one recently uh this notion of lake effect snow i'm from buffalo new york and when the storm happened last year um i i grew up with lake effect snow i was at five years old in a blizzard of 77 you know and i know what it's like to wake up and the snow is higher as high as your house you know that's what i made all the money as a kids you know digging people out and all that stuff but when you know but again if somebody's from out of town or if somebody is is new to the area that doesn't mean a whole lot what does lake effect snow mean well you know 52 or so people died were probably people who understood what lake effect snow was um and i guess in a way there's this this idea of semantic satiation where if you hear so much it kind of loses its meaning you don't really react to it so i think going to uh jose's point uh you have to have people at the table who can help you hit that plane speak you know that right tenor with the messaging because even if you have people who are familiar with with systems sometimes that familiar language is what lows them into a false sense of security and i see that in florida all the time so um the other the other word that i see uh sort of lose its value is more accessibility means different things to different people um a person with a wheelchair accessibility means one thing that differs for somebody who needs a sign language or who needs 508 compliant you know websites from the county to understand the uh nature of the threat so i think accessibility is another word we can do a better job of parsing so that we don't uh underestimate all that resonates with people and it may miss the mark if we're assuming that it means one thing and i think sometimes we often use it that way uh a little too much thank you both are there questions from the audio okay andrea i see your hand first and then mickey um this question is for for you sherman um a lot of the issues especially for exist for instance the example you gave us with the fire and your own decision-making process a lot of that seems necessarily less to do with communication and more to do with resources you were making a calculus that you know i don't i think was okay i mean it was based on your own experience and your own needs and i think that's probably true in a lot of cases you know i heard stories from hurricane and a lot of elderly you know have different risk calculus is when it comes to going to a shelter for the same reasons because am i going to have to sleep on the floor i'm 80 um and so i guess the question here is how much do we focus on communication versus actually bringing in these better resources and then communicating that we have those resources to get um to to really provide for people who need i don't think it'd be it can be an either or proposition i think you almost have to do both because communication comes in layers you know it's not FEMA but not the first messages it's you know a lot of decision makers at the state and county levels the sheriffs you know all those people they're the ones that are going to decide whether they're going to make you know release units available to come rescue somebody who who underestimated how fast the flood water was coming in or how bad the storm surges were going to be um so they're gonna have to live with the consequences of it so oftentimes they're the ones that know the community the best but the resources does become the thing you know i i had resources but i still did not feel like i was going to be in a better position than if i just waited out and maybe um you know let's see how bad this will get before i really have to take that step looking back on it it wasn't a great decision only because i know how bad it can get now if i never had this job i probably would have thought um you know good decision and i probably will be in another wildfire and make the same mistake and it could be again at my peril and that's part of the problem is people think that you know because they've gone through 10 hurricanes that this won't be different until it makes that turn and washes away parts of the country and a lot of people don't want to hear about climate change but there are weather systems hitting areas of the country where it didn't hit before and that's a reality and so you've got places that are not prepared on top of the fact that people are under resource so um it could have a cumulative effect but i think that's why the best opportunity we have is to have people sit and even if it's not adequate talk through it like give people an understanding of what they're going to be up against and and then hopefully um they'll find the resources if they you know if they're not provided they'll find a way to prepare and access those resources mickey thank you to you both that was really powerful hearing from both of you so i have a policy question um so what pressing policy do you think your agency needs to create or implement that would best help these populations that you mentioned so sherman if you could say what you think you much to be doing joseph i know you don't formally work for no or the weather service but feel free to say what you think they should do so perfect thank you sherman and thank you mickey no yeah honestly i think really just fundamentally we need to first even define what a translation means for an agency one of the very first issues we ran upon when we started embarking on this bilingual risk communication work is when we're providing these recommendations agencies would say no great this is it's great to see that you know the data shows that these words are resonating more with the community we'd be happy to change it but wouldn't it be a little troublesome if the english and spanish didn't line up necessarily and well like when you really think about it when you're bilingual you realize that words technically aren't directly translated all the time and that that was one of the very first instances where i thought well like we should really just define what a translation means for a lot of these agencies because that's probably why we're having these issues of creating different variations of translations is we're too worried about directly translating the word rather than the meaning of it overall and so i think that would be a fundamental first step for a lot of governmental agencies to consider what does translation mean and after defining it actually start to implement some of these things because if we are more concerned now about translating the meaning of of given words overall then yeah we'd be able to move a lot more quicker when it comes to actually creating words out there making sure they're consistent and most importantly making sure that they resonate with the community one of the biggest i'll call it a challenge one of the biggest challenges coming into fema and doing the work that i do was first having the culture of the agency understand that there are certain points in the disaster where if you wait too long it's too late and because we're a reactive agency by design i mean there's a there's a declaration process for a reason states have autonomy tribal territories have autonomy you know so you can't just have the feds come in and rush in and and take over um but there does become a point when there are diminishing returns to the communication because you can you've you've the people have no decision-making capacity at some point and they're they're at the mercy of the of the circumstances um so one of the policy changes that i sort of uh you know um inculcated was the idea that for the disability population errant this is the first response and that means that during that pre-land fall period even though we're not in there you know urban search and rescue is getting staged and all the things are happening we have to begin to profile the risk as accurately as we can anticipate that there are going to be things we don't know um but try to be as certain as we can and have relationships with those community stakeholders and empower them with enough information to sort of set conditions so that we'll be successful when we come in i'll give you a good example is up in the northeast right now you've got a lot of flooding in areas where people aren't used to it and one of the questions i had for the state um was uh are you alerting people who are on uh who are electricity dependent about the nature of the rolling blackouts that are going to be necessary you know now can FEMA do we can't we can't necessarily do that per se but i can tell you if that's not if that communication isn't going out to those people you're going to start to have people in dire straits that would have been uh mitigated though had they had information ahead of time so i'm always looking at blackout situations as an opportunity and i'm not talking about where it just happened like a tornado i'm talking about where they're going to have to shut it off to do certain work and we know how many people are electricity dependent you got to put something out so i think that that more proactive posture is slowly seeping in and i'm making my way around all the program offices because again we're a reactive agency by design but in some cases we can't be reactive or if we are reactive it's an anticipatory reactive you know that's kind of a you know um kind of intuitive but if we anticipate a threat um we're already probably too deep into it to react to it but we have to begin to communicate to people and give them information so that they can make the decisions all right thank you so much i think we need to take a break now so thank you again for a week