All right, so welcome everyone. I'm so excited to be here with you to do a whirlwind tour of user experience techniques. I am Laura Quinn. I am a independent consultant, a specializing in research, so both user experience type research and evaluation type research for access to justice technologies. Many of you may know me from my prior life as executive director of Idealware in which I also work with a lot of legal aid organizations and other access to justice organizations. So I've been excited to bring kind of a lot of different pathways in my life together to specialize specifically in access to justice. And with me today, I have two folks who are going to both be, they're gonna be presenting PACE studies as we get towards the end. And they're also going to be doing what we call, we're calling color commentary. So they're going to be joining in to provide a little bit of additional insight on it. So it's not just a wall of me talking to you. So we have with us Rachel Harris. Rachel, you wanna introduce yourself? Yes, thanks Laura. My name is Rachel Harris. I work at the Ohio Legal Assistance Foundation. We are the state funder for Ohio's legal aid. I am the project coordinator for our Ohio Legal Help Project which is our statewide portal project. And we have been working with Laura, especially over the past six months doing a lot of user testing in anticipation of the development of that project. And I'm really excited to share some of what I've learned with you all today. Dan, fantastic. And we have also with us Dan Jackson from the new law lab, Dan. Hi there everybody, Dan Jackson. I'm the executive director of the new law lab. We are an interdisciplinary innovation laboratory of Eastern University School of Law. We've been around for about five years. One of the things we've been doing in those five years is working with some legal aid organizations among this community on some of the tech tools for self-represented litigants. I think folks might be familiar with our game, represent, two games now, represent, represent, return that was with Connecticut, statewide legal services of Connecticut. We also collaborated with Pine Tree Legal Assistance of Maine on their state side legal initiative. Thrilled to be here. Fantastic. Great. All right, so what are we planning to cover today? So we'll start with just a quick introduction to what user experience is. It's kind of a jargony term, but I like it because it kind of captures an important essence of what we're thinking about today. And then we'll be thinking about it in kind of three categories. Defining your audience, understanding how they view the world, and then getting their reaction. So thinking in these blends a little bit, they're not pristinely distinct categories, but we'll be thinking about what techniques we can use to do those three things, and then also outputs so that we can share that those findings with others. And we'll be winding up with two case studies as we talked about to kind of show how these actually come out into the world, which is often a combination of things. And throughout, we'll be focusing on fairly affordable ways to do things. So assuming you don't have an enormous budget or a ton of time and money, what are the important things to keep in mind here? So I really do wanna pause here. It's gonna be a little interesting. I hope you'll help me out. Sorry, I do wanna pause here to ask you what it is that is particularly interesting to you here. So if you'll take a minute to enter something into the questions page that is kind of a question that you have or one of these that seem particularly interesting. And then I'll ask, sorry, if you can read some out to me or the themes, that type of stuff, I'd love to get a sense as to where people's mind is at. Well, test your multitasking by starting to read while some of you are still typing. I'm gonna just pause and let you type. I actually got the questions down going again. Most of the questions so far have been just me helping people out with audio. Okay. Oh, come on guys, you can't leave me hanging here. I'm interested in the on the ground logistics of UX testing. How to do it with limited staff and budget. Fantastic. Evaluation techniques for self-help portals. Great. And I like to just pause here until more people give me some input here. Yeah, we've got somebody from the Massachusetts tribal courts here who is interested in how you are getting reactions from desired audiences. Great. Great. Guys, certainly feel free if you're still typing, enter it in and we'll capture it. Yeah, so it's great to know kind of what you're interested in covering. So perfect. Let's dive in here. So what is user experience? So I'm using the term user experience, which by the way is often abbreviated UX. So if you see UX, that's what it refers to. And the sum total of your audience's interaction. And because we're talking about technology specifically, it would be the sum total of your interaction with your technology. But for some, like if you're thinking about it for your organization, you could think about the user experience with your organization. And that would be all of the different ways that they can connect with you, how the conversations that they have with your lawyers and how they feel about those. So and for it to be a great user experience, I really like this definition. It should meet the exact needs of the customer without fuss or bother. And that sounds like a really simple definition until you really start to think about it. So it implies we need to know who the customer is, number one, and customer, I think we can also, we can say, you know, constituent, consumer, you know, whatever word we wanna use there for someone who'd be accessing, access to just the stuff. So we need to know who they are, the audience is. We need to know their exact needs. And that is certainly, I mean, I honestly will never know their exact needs, but we can get as close as possible. And we need to be able to figure out what is going to meet them with, by their definition without fuss or bother. So it's actually a fairly powerful definition, I think, of what it is that we're trying to do. And the main technique, or one of the main techniques you have to get there is to learning from those in the audience. So basically we think about ourselves as apprentices to actual individuals in our audience who know far better than we do what it is that they need, what it is that they do. That's not to say, as we'll talk a considerable amount about, that's not to say that if we ask them, what do you need? They will be able to articulate that well, but they certainly, the information about what they do and what they would use is much more in, it's more in their head than it is in yours. Yeah, there's another good one here, which is, do you ask users if they had success in their court using law health resources? How do you measure success for users? Kind of the difference between user experience and judging outcomes. Fantastic. Or interacting with those two. Yeah, I will, in fact, that's a great question that I didn't actually have anything to cover, but I'll put that in as we talk about getting people's reactions, kind of the idea of reactions and success. Absolutely. All right, so we're gonna think it through in three parts as we talked about. So thinking about who is in your audience, so what are their characteristics? So kind of some basic definition of your audience. We'll then go, we'll think about ways to dive deeper. How does the audience currently do what you are trying to support? They do it somehow, even if it's by talking to their barber or filling it out in paper or there is some method by which they are at least attempting to accomplish what you are doing. And how they think about it, which can often be very different about how we, as experienced technologists or lawyers or both, are thinking about it. And then getting their reaction. So basically, so we're understanding how they view the world before we necessarily have anything to put in front of them for them to react to. But then it's also very useful to put things, so our own design for a technology system, somebody else's design for a technology system, an actual working system, whatever, in front of them to get their thoughts. So we'll talk through that. All right, so let's start with defining your audience. With the idea that there are many, many audiences that you could define. So the idea of the dartboard here is not that we're just throwing darts to find your audience, but that each of these, you know, we've got all of these, the different segments, and there's overlap, and this is often a really complicated thing to think about as to who your audience is. So in this realm, we're thinking specifically about who the target audience is, how many people are in that. So we know whether we're designing for a very limited audience, a big audience, and kind of what the difference between those are, things like their literacy level, their tech proficiency, general trends of their needs. So we're gonna, in the next section, talk more about kind of their, much more about their, how we would understand their needs. Or we're gonna go through each of these in more detail. We're gonna talk about quick conversations with subject matter experts. Sorry for my abbreviation here, SMEs. Talk about review of existing research, surveys of target audience, creation of personas. So basically a bunch of different ways that you could understand who is out there. Simply with the thought process, which can really be instructive and provide a lot of guidance for you. Simply by saying, okay, who specifically is our audience? What is the defining characteristics that define them as their audience? And who isn't in that? And if we have a number of audience, because it was often true, how can we prioritize them? So basically starting to say, all right, everyone is not a target audience. Also low income people in Ohio is not really a feasible to work with target audience. We need more specific parameters than that in order to really be able to start to think about who we're working with. Dan, I'm gonna put you a little bit on the spot. Do you have, as you've been working on, kind of all of the different projects you've been doing, have you, do you have any specific techniques that you've used to just start to think about how to crystallize out what audience you're even going to talk to, test with? Oh, well, it very much depends on the project or the problem that we're trying to solve. I mean, our work really is focused on working directly with intended end users. So it's co-design and other ways of working directly with folks for whom we're designing. And so in that instance, of course, the user's already already defined. And so it's very much based on the problem that we're trying to solve. For, you know, when we worked with the statewide legal services of Connecticut, we had self-represented litigants, obviously it was the chief target audience. And so we worked directly with those folks in designing the game itself. But there's of course lots of different, you know, subsets of individuals within those categories. But we let that process, we let those folks to sort of define that themselves. We tend to err on the side of giving our users the tools to direct the process. Yep, absolutely. Yeah, so basically a, so starting to think about this is simply a more complicated than it might seem. Like to sort of assume that you may have some questions about, as well as answers about who your audiences are. They will want to explore further research. Speaking of research, thinking about academic research, we are lucky in the legal aid world and the act of justice world. And that there is in fact, often academic research that overlaps with topic areas that we're interested in or the target demographics that we're interested in. So if we are, for instance, you know, I just, I heard Sarton mentioned that there's someone on the call who works with First Nations tribes understanding how First Nations folks think about access to justice or they think about what it means to get a fair day in court. I have no idea whether those studies have been done but it wouldn't surprise me. And that would be really informative in that type of work. If we're doing technology work to target an audience like that. So thinking through demographics, characteristics, including like as you just try to get your hands around it, understanding what other people thought was important to understand is can be really useful and some key themes. So I've just pulled a couple of things that I knew existed just to give you the examples of the types of things that do exist. There's a very big piece of research that pulls together eviction statistics that has a really nifty side called the eviction lab.  The IELS, out of the University of Denver, I believe, did a research on experiences of self-representation in the U.S. Family Court, which is a fantastic report. I've just pulled, I was thinking about disasters. First I had a different project I was doing and I'm like, well, I bet there's stuff for, about how we help people who are in the midst of disasters. Did a Google search and two minutes later I have this report that would be a really interesting piece of this. So basically the idea being that these are resources that really are often much more useful than it feels like, I kind of feel like people assume they're not going to be valuable when in fact I find that certainly at least this background reading is nothing else. They are more likely to be useful than not. And we'll talk about some ways to use that in a second. Analytics data, so this is something that most technology people on the call are probably familiar with from potentially a different perspective. So thinking through what things like your Google analytics on your or somebody else's website can tell you, social media sites also can tell you things like age and gender, can tell you what topic areas are of interest to people, can tell you how they tend to navigate through websites, things like that. And with our, us being a very close-knit community in the access justice community, keeping in mind that often people are willing to share. So if you have somebody else who has a website that seems like it might be targeted at a similar demographic, similar audience to say, well maybe there would be worth, would be willing to tell us some of this information about their own website to the extent that it's useful for us. So we're gonna talk a lot in the next section about interviews with target audience members, focus groups, things like that, which would be certainly my preferred method. But if budget does not allow for that, just simply talking to people who are out in the field, so line staff who work directly with your target audience can be really helpful. So both in a legal aid context, so this might be like talking to your lawyers who are actually working with you, directly with your constituents, but then also saying, all right, well we're interested in thinking about issues of housing. Well, let's go talk to some people working in homeless shelter about what are the typical issues they're seeing that have gotten people there initially or preventing them from getting back in housing, or that type of thing. So to think through, if we don't have the money to go talk to a bunch of people of direct, sorry, a bunch of people who are actually in our audience, we can at least go talk to people who have directly interacted with a bunch of people who are in our audience to try to get viewpoints there. And you wanna do at least three to four people. So in general, as you're thinking about interviews, so one is not enough because you can have just a really weirdly skewed perspective. Two, you have the potential that people really disagree with each other and then you don't really know where to take it. Three starts to be, you could start to build up some themes. Four is, then you can begin to feel what's okay. Everybody's to all four people said this. This is really important. This is only one person said this and it seems pretty far field. That's we don't need to care as much about that. This conversations with subject matter experts. Dan or Rachel, I feel like, I feel like I'm talking a lot. So I'm hoping that one of you, anything to kind of add on the idea of the possibility of talking to or including kind of subject matter experts either in addition to or opposed to myself. I'm sorry. Yeah, so this is Rachel. Yeah, so I think another element of this that was very helpful for us in our user testing was we started out our project with some quick conversations with subject matter experts both in our legal aid organizations but also in other community organizations, community development corporations and community action agencies which administer certain public benefits in Ohio. And that actually ended up being a great connection to have established later on when we were looking at our user testing and looking at locations to do our user testing. And looking back on it now, we realized that that really gave us a great opportunity to kind of start looping others in on our project. So it kind of had double value and it was very great. Yep. The only thing I would add to that is be respectful of your subject matter experts time and therefore, you know, schedule that opportunity to speak with them at the right point in your project so that the ideas are starting to come together in such a way that you're actually gonna be able to have a real focused conversation. It's not always helpful to have real meandering and vague conversations early on in the process with the subject matter expert because they may not have anything to add value there. That's interesting. Yeah, so I would say certainly have a focused conversation, no questions. I'll have a focused list of questions and we'll talk more about this as we get to interviews. I would say that it doesn't necessarily need to be later in the project process, from my perspective, because we might ask them really focused questions like for instance, what are the, if I'm just trying to think of an actual example of a technology, let's say we're building a volunteer portal for lawyers. And so we go talk to, the obvious line staff would be the people who are working with the lawyers. So what are the most typical obstacles you see to people actually, so someone has expressed interest, what are the most typical obstacles that keep them from actually doing something? That's a very focused question, but it's very early on before you've actually defined precisely what the project is. So, absolutely, great. Surveys, surveys are really interesting. I feel like people often go to, I think surveys are people's go-to for getting information on what people want or need. And I feel like they're definitely a useful way to get answers to a few questions from a lot of people. They will get you shallow information but very broad information. However, they're harder than you might think, especially to get reliable information. So unless your group is very discreet, so unless you can fairly easily define everybody who is in it and have some reason to expect that you can get a fair number of them to answer your survey, it's hard to get a sample that actually, it's hard to get the right people to answer the survey or not the right people, but a selection of people to answer the survey that actually you can have some confidence is not skewed in some weird way. So basically, unless you can define why it's not skewed in some weird way, you need to assume it probably is. You can say that, all right, well, this is interesting information that is, can guide us, but unless you've actually talked to someone, hired somebody about getting a representative sample, which that typically takes time and money, you're going to have to assume that it's gonna be for guidance only, and they need to be written carefully as well. So the questions needs to be carefully done in order to not sway the audience. So in fact, the further I go in my career, the less I tend to like surveys as a quick and easy research technique, particularly, I think, because they seem quick and easy, but they don't give you data that is really easily actionable. So personally, I would almost always go talk to five people then survey a broad-based group of people and just kind of see what happens mostly because that broad-based group of people will not give me a lot of survey results and I won't know what can make of it when it happens. Does anybody want to disagree with that? That's actually a fairly controversial opinion. Anybody through the chat have different experiences or Rachel or Dan want to disagree with me? So I think I strongly agree that surveys have some challenges and it's even a different, there's an additional set of challenges. When doing user testing on web resources or other things, watching someone interact with something versus self-reporting afterwards can be fundamentally different. We are not the best at reporting on ourselves. We're often a bit self-conscious and tend to try to defend our ego when filling out a survey. So watching someone and having that actual user interaction where you see them interact, you're going to get a much more honest, unbiased understanding of how your resource works. Yeah. Totally agree there. I totally agree with that. Great. Perfect. All right, let's talk about some ways to actually, so we've gathered information about our audience. What are some ways to put it out in a world in a way that people can kind of process it? So it's not just a bunch of data. One of the ways that I've found to be most useful is a topic or a technique that's called personas, or is this technically person-y if you want to get really technical because persona is a Latin name and the floor would be person-y. Anyway, so this is basically, we've got a fictitious person who's serving as kind of an archetype for us to be able to focus our design energies around. Because it's a lot easier to think about how we would support Susan, who is a supportive social worker, than it is to kind of think through the general idea of how social workers want to use the site. So we could say, would Susan like this site? I'm sorry, we wouldn't say that. We would say, would Susan use this particular feature? What would Susan do here? So the reason I just backed out of would Susan like this site is because we tend to, as Sarge just kind of mentioned briefly, we try to stay away in user experience from the idea of what people actually like in favor of what they actually do or use because people will often self-report that they like something that does not actually save them time or increase their understanding or lead to any actionable goal. This, what we're looking at here is a, so this is from Illinois. It is using a template that you can easily find online. If you just search on Persona Template, you will find this template that kind of helps you to think through, all right, here's, I'm gonna just kind of plug in, all right, here's her goals potentially, sorry for motivations. So this reads, empower clients, close cases, receive recognition, a bunch of goals, challenges. Here's another one. This is one that we used for Ohio Legal Health. I kind of like, so that particular template we were just looking at, for me it has a little too much information. I kind of like a narrative instead because it allows you to kind of cut to the chase a little more quickly. And yeah, and so you can see, like for instance, in this particular case, we also sliced the Persona somewhat differently. We've got Matthew watching the big picture. We also had somebody who just wanted quick answers and we had someone who really wanted support, wanted someone to tell her the answer. But things to include in these are context so we can really feel like we can get to know our Persona a little bit, how he's accessing it. So for instance, Matthew here, he doesn't have a computer or a smartphone but he uses the library, the computer. Doesn't use the internet a lot for every little thing but he can generally get by when he needs to use it. And one, it basically is defining characteristic is he wants to understand all the options. But like in fact all of our Personas for this, he's little knowledge of legal terms or potential courses of action, not a lot of experience with researching things. So these are the things to include to help make that useful. Questions or thoughts about kind of this whole idea of defining your audience before we look a little bit and understanding how they view the world? This is Dan, I would just throw out here that always include some extreme users in your portfolio of Personas because that's a great way to make sure that whatever you're developing is gonna be able to hit all of the needs of all the fast, all the different folks out there with very different needs. Extreme users are sometimes the ones who are most likely to break it and therefore you wanna have that in mind. Yeah, and there's actually lots of schools of thoughts about this. There's actually, we could probably do a whole session on Personas because there's a school of thought that says you might want to focus in only on like one or two extreme users, particularly if they seem like they encompass other folks. So to say, all right, well the person who is not very literate, who is using it on a phone, who wants to understand every option conceivably and it's gonna be entirely dependent on the project, but conceivably that might encompass a lot of other Personas and we can laser focus on that one, which would be really handy if we can. All right, let's think about and diving a little deeper into understanding how they view the world. I've purposely got this, this is a upside down map from the perspective of Australia. So thinking about this, this is often kind of what we learn when we talk to people on the ground. Like the facts are the same but they're perceived in such a different way to not really be possibly even that recognizable. And so it can often be really transformational to go talk to real users and get real people involved in your research and your project. So here's some key things that we're going to, that you'd wanna look at here. So what would motivate them to use the system? So you wanna think through their motivations in general and their motivations around the system. So what are their goals? I think it's easy for us to focus in on our own goals but fundamentally if our goals don't intersect with their goals, then it doesn't matter because they're not gonna wanna use the system. And how do they currently achieve their goals? So they have a goal, like let's say their goal is to understand how to get a divorce and they are achieving it in some way. Even if that's going to talk to their cousin Larry who really has no idea about how to get a divorce and gives terrible advice, they are achieving the goal of understanding this. They're not understanding, it's not a correct understanding but they feel they've accomplished it. So it's really important to think through their current processes and understanding kind of how they think about those related processes and what works well and what are the gaps. Some of the same things that we just talked about would work here. Nothing else. Quick conversations with people who actually work with folks on the ground. So you'll hear the theme throughout. So best to talk to the real audience. If you can't talk to the real audience, at least go talk to people who work with the real audience. Insisting research can be really useful. We'll talk here about interviews, focus groups, surveys are again interesting and we'll talk about documenting these through mental models or process flows. Interviews. So this is a great way to understand context. This is probably, this is my, honestly my go-to as a researcher at the beginning of a project to kind of understand what it is that how it is that people are thinking and to establish kind of the overall context for whatever it is that I'm investigating. So you wanna write out what questions you'll ask and as Dan mentioned before, you definitely don't wanna make it feel like a fishing expedition. You wanna make it feel like you are asking questions that they will know the answer to. So it's not just the general, there's three questions in a half hour interview. It's more like there are 15 questions or 10 questions in a half hour interview. You wanna write them out but you don't wanna actually read them verbatim to sound really scripted. You wanna make it a conversation. We're gonna start with easy questions. So don't start with a really world-changing question. Almost anybody that you talk to will say that they don't know enough about whatever you're asking them about to offer an opinion. Strangely enough, actually I find the more experts they are in that particular area, the more likely they are to tell you that they don't actually know anything about it. And it's certainly true if you're going out to talk to members of your target audience that you wanna start by thinking about making them comfortable, establishing a bit of a rapport to make yourself seem approachable and not kind of in an ivory tower. And I actually find a really helpful mental model for this, for myself. It's thinking of myself as their apprentice to understand what they think and do. Certainly no one understands what they think and do better than they do. And so, and that is all I'm there to do, is understand what they think and do. So that may or may not be helpful to you but it's always been helpful to me. And again, three to four people for each demographic you care about, this can actually get fairly complicated fairly fast because if we say, all right and we feel like we care about both people who are living in urban areas and people in suburban and in rural and we care about high tech literacy and low tech literacy, this tends to, the typical way to do that is to form a matrix of kind of thinking through all right about how many people would that be and either carefully or approximately trying to segment. Rachel, any thoughts on interviews? So in terms of starting with easy questions, that was when we were doing our user testing, one thing that I wish we had thought about when we were writing our script is we did start with what we thought was an easy question which was what you do. And we were testing with low income users and many of them were not employed. And we quickly realized that that was just totally off-putting and not a pleasant way to start the conversation. And so we altered that a few interviews into, who do you live with? What's your family like? And that was much more effective and much more conversational. So just thinking about the cultural context that you're in when you're doing your interviews is something I wish I had known. Absolutely, absolutely. Yeah, the only thing I would add on that is interview questions, it's a tricky balance because you have to be detailed enough to get what you need, but at the same time, if you're too focused, you're gonna miss out on a lot of potentially valuable information. Especially for lawyer designers, I think we can sometimes get a little bit too detailed and it's not a deposition outline. You wanna be thinking more along the lines of broader questions that will allow people to really explain how they might interact or how they might use this product. Yeah, yeah, absolutely. Fantastic. Focus groups. I'm gonna actually move a little more quickly through things because I'm realizing that I am, I don't wanna take away time from our case studies here. Focus groups are, in my mind, generally quicker, but often not as good as an interview, especially if you're not a very skilled facilitator of focus groups, they're harder in my mind to do well. So again, you wanna write things out, make it a conversation. Groups of about five to eight enough so that if people cancel, you'll still have a group, but not so many that you've got overwhelming. Helpful to have exercises or an exercise that the groups can do together. You wanna think through who you're having in the room so that if you can, to try to avoid having one person or two people that everybody else will defer to in the room because it basically, at that point, becomes an interview. You've just got a bunch of people sitting in the room for it. And it can be actually tricky to figure out how to, how do you take notes, how do you record it, to make sure you think about that in advance. Dan, I know you mentioned that you had some experience here. What would you add here? No, just that I wanted to point out with focus groups, I think they're most successful if you're working with a wide range of stakeholders and you're gonna do focus groups, I would keep or consider, at least, keeping your focus groups to one discipline or one subgroup of your stakeholders because you really do end up with, it's very easy, especially if you've got lawyers and non-lawyers in a focus group, it's very easy for the lawyers to dominate both because lawyers tend to do that, but also because the other folks around the table, if there's a point of the focus group has to do with a legal issue, they will often defer sort of automatically. So I have, and when I've used focus groups, we've tended to, you know, we're gonna have judges, we're gonna have all judges together and then we'll have self-represented litigants separately. Yeah, great, I'm so glad you said that. I don't worry, you're going the opposite direction. No, I have 100% agree, sounds great. And keep in mind that recruiting is often, for all of these things, for interviews, focus groups, focus groups, recruiting may be as hard or harder than actually conducting them. So figuring out where are the people coming from, how do you get them in a room? So ideally, if you're doing something relatively short, so you're doing like a short interview, go to where the people are already. So basically show up at a, I mean, it could be many places. You probably wanna go someplace where you have permission to do it, but you could go to a, like Rachel mentioned, if you need the action agency. And just basically see if you can find people who are willing to take 10 minutes or take 20 minutes or even take a half an hour to talk with you or do a user test. Incentives, things like a $10 gift card seem like, so in my experience with a couple of user tests with this kind of low income population seems to be sufficient. In fact, things to be popular. And remember with your doing appointments in advance that many people will cancel. And you can't, you certainly don't wanna do things like double book yourself, which would be really unfriendly. But remember that that is a possibility, that you don't wanna assume that 100% will be there or you'll get half the people and it will go all right. Rachel, you wanna add just something quickly on recruiting? I know that you guys gave a lot of thought to recruiting. Yeah, so we were very fortunate that we had good relationships going in with our test sites. I mean, as you said before, you might wanna pick a site where you have a good relationship with the staff already, that will make it easier. And another kind of tip that we had was having the staff members who theoretically, at least for us at the community action agencies, the users were already very comfortable with and associated them with receiving good outcomes. So when the staff members asked them to participate, I think they were much more comfortable versus me, a stranger in a suit coming up to them and asking them questions. And so that was very effective. Great, fantastic. Just a couple of thoughts about what to do with the data. So you typically for, so an in-person or all over the phone interview or a focus group or a user test, so for all of the things that we'll be talking about, you typically take handwritten notes and then you very soon afterwards try to write it up so that you don't lose like handwritten notes you tend to forget what they mean very quickly because everything starts to blend together. So you will typically write up the notes and then you will go through and code them. So like for instance, you'll have a statement and this statement was by this person or this participant number. And you'll put codes on it to simply say, all right, well, this one I just decree that this one is about lockout, at least three are about lockout. And then I'm just going to like maybe for this particular set of data, I feel like with one category is insufficient, I'm going to add another one about not knowing who to call so I can go on it kind of a different way. And then I can quantify this way. So I can say, all right, 80% of those illegally evicted simply moved on without taking any action. And 36% specifically mentioned that they didn't know who to contact to preserve their rights. So this is completely legitimate data. This is something that is done in academic research all the time. It's time consuming. I generally estimate this at about an hour for each interview hour. So basically a minute per minute on the actual time that it took to conduct the interview to then analyze it and quantify it. Here's another thing that you can do with this. And sometimes you get a lot of data that it's hard to quantify.  This is a diagram that's sometimes called a mental model diagram. Rachel is going to actually talk about this specific diagram. And in fact, I noticed she picked a very specific screen, very similar screenshot. So I'm not gonna actually talk you much through this. But the idea of it is that basically I am documenting the general trends of what people in this particular screenshot are saying that they will, how they use internet research. So they'll quickly scan the page, look at it and see if it looks credible and useful and hear some things about this. Quickly scan to see if the info I needed there. This particular one was actually created based on academic research. So we never actually spoke to anyone for this. We used like five academic pieces of research which in total had like 200 interviews. So it, more than that in fact. So it summarized a lot of different research. Another thing to explore further if you have a lot of this kind of data, it's something called a customer journey math. Which is another kind of diagram that is the term is used a lot for a lot of different things. But just in general it refers to mapping out a process that's either kind of a process in time or in space or both. And then kind of what people are thinking and doing during that. So this would be another term to kind of look off if you're thinking about doing diagrams. I find a mental model is easier because of the bunch of boxes.  This one tends to look more like this which kind of requires a little more fancy visual design field. Questions, thoughts on any of that stuff before we talk about user testing. Getting their reaction. So just our kind of quick overview here. So now we're thinking about how does the audience use? Oh, well here's one that we'll actually look at in more detail. How does the audience use other people systems for doing what we're talking about? How do your designs work in real life? How are they, how well are they able to understand and use the design? And last and I would say least what suggestions do they have to improve the design? And so I feel like when you're doing user testing specifically and this actually becomes a little different and we'll talk about co-design sessions as part of this. A co-design is a little different. User testing you're generally focused particularly on what it is that they're actually doing as they test. And you can think here through testing your site, testing other people's site, testing people prototypes and co-design sessions which will define. All right, user testing basics. User testing I would say is the opposite of surveys. It is much easier to do than people think and it is harder to go wrong. So it is worth trying. There's definitely, you want to think about what you're doing. So you want to practice a little bit. You want to think through best practices but there's really no, there's no harm in getting actual audience feedback on things and things I would say the earlier the better. So there's no point in designing and building an entire website and then testing it and finding that there's a lot that you would change about it except you're supposed to launch in two weeks. Like that's just a waste of testing to me. So you want to try to think through how you can test early in order to get that feedback. So let's see, so four to five people for like a half an hour scenario. So generally a user test is about a half an hour so I think it'd be less than half an hour. In general you'll get a lot of the feedback you're going to get. Like I think there's someone, there's a lot of studies here that say actually like 80% of the feedback you're going to get people can get in four to five people. You want to script out the test carefully. So you want to write down the questions you're going to ask. You want to think through how the flow is going to work. You want to try it out because especially if you're working with folks who are less internet savvy there's kind of a power dynamic between you and them. You want to make sure there's not just kind of a weird awkwardness in the middle of the test. We're just going to throw off the whole dynamic and the feedback. So you basically ask them to take the computer and do what they would do if you weren't there. Give them a scenario and ask them to think out loud. So basically ask them to talk a little bit about why they're doing what they're doing is the general sense. And it's useful to try to give them a real scenario. All the way to if you can, asking them to think of a time when what you're doing would have applied to them and they can, you know, come up with the, they can come up with a scenario that they would actually walk themselves. That's the ideal. I find that it's often tricky. At least give them a scenario that they can kind of put themselves in and kind of play ask through. Recruiting may be tricky in the same way as everything else. So this is, to my mind, it sounds intimidating when I talk through it, but in fact, it is really not that hard. And here's if you Google this, or we could send the link around. Rocket surgery made easy by Steve Krug. This is a book, but he has put out a usability demo, which is just him on YouTube. It's just a video of him doing the user test. You can see how it really is not all that complicated. Rachel, do you want to talk for just 30 seconds or so about kind of what you found? Let's go with the positive side. What's your found easier than you expected about user testing? Yes. So first of all, I, you recommended practicing beforehand. So what I did was I practiced with one of my colleagues that I was very comfortable with and I recorded it and then listened to it later and also took me back from Laura and my colleague. And that really put me at ease, especially once I realized that as I got more comfortable in the test, it went much better. And so I was like, oh, well, if I just relax, you know, I don't have anything to worry about. It was also, you know, kind of a situation where I was afraid that people would ask me questions that I couldn't answer or I wouldn't be able to, you know, inspire confidence in them. And then, you know, I got there and I realized, oh, well, first of all, I've been working on this project for a year and a half. So that's, you know, I am the expert. And secondly, people really felt like we were listening to them and we offered the $10 gift card incentive that we talked about earlier. And just with those two things, people were very, you know, positive towards us. And so I think I was a little intimidated and I thought people would, you know, question why I was asking them to do things or kind of give me resistance. And I think just by relaxing and realizing that, you know, not only are they doing me a favor, but I was doing them a favor, it made the whole process a lot more comfortable. Fantastic. Great. I mean, so, so capturing the data, so something that people often don't think about until too late, if you're going to, like, so if you're only doing four or five, you can probably just understand what the issues are and revise. If you're gonna do a larger set, because for instance, you are testing for a bunch of different things, then you are, yeah, a big bunch of options compared to each other, you probably want something like a spreadsheet like this to write them up after, so you can see things like, what path did they take through the site, comments or issues or good things or bad things, et cetera. So different things that you can test. Here's another thing that I think that people don't think about nearly as much as they could. There's no reason that you can't test things that other people built. If it is public and it is live on the web, then you test it. So like for instance, this is a screenshot of I think this Virginia pro bono portal. If I were, and I just arbitrarily selected that, if I were building a pro bono portal, there's no reason that I couldn't test a number of them to understand what works well and what doesn't work well. And this is something that's done in the competitive world, sorry, in the corporate world all the time. It's called competitive testing. We don't compete in the nonprofit world, so we don't do that. But if it's there, I mean, certainly it's polite to tell people you're doing it, but there is, I mean, it's public, go for it. You can also do paper prototyping. So literally creating a mockup in paper of what you are thinking. You can see here that there is an interface which is designed in paper. This is actually a fairly complicated interface. And ask people to like, I will, I've done this before where I told them, all right, I'm giving you this magical clicking stick, which is like a pen. And it's like basically you just, people tend to be, they think it's fun, it's silly. And you actually get a lot of good feedback because it's so obviously not final that people are willing to give you feedback. Obviously you can test your own system. So if you are improving an element of it, so test before you plan to refine it to see if there are additional things you can add in or test as you build to validate that people respond, to expect them to respond. Co-design sessions are an interesting thing. They're kind of relatively new to the world of UX. They're kind of a combination of a focus group, a design session, and a user class. And actually I was going to ask Dan to give his perspective on it. He's a lot more experienced than I. And because we're writing short on time, I'm just gonna hand it over to Dan. Dan, can you just talk a little about what a co-design session is? Sure, yeah, it can take so many different forms really, but the way that we try to deploy it at the new law lab is through sort of convening core groups of co-designers, stakeholders over the course of the lifeline of the entire project. So that you start with those folks when the ideation and prototyping and the testing and you're doing that and regular intervals which is going to not just create the, iterate the actual details of the product specifications, but it can also help transform stakeholders into co-creators who aren't just there with buy-in, but also have that sort of pride of co-authorship. And that goes a really long way with marketing and getting your tool out there. So co-design, at the bottom, it really is working directly with the intended end user in a collaborative way to design the product or service itself. And again, we deploy it in a lot of different ways and there's a lot of variety as well. The way we did it in Connecticut with the game was through, like I said, convening stakeholders, having a game design workshop that was a lot of fun, and then reconvening those folks to test the actual first iteration, the first beta of the digital game. And then reconvening again when we've got another grant to do another game that was focused on eviction, and that sort of created sort of this game design studio, if you will, down in Connecticut. Yeah, yeah, so they're kind of, if you envision them as a combination of focus group design sessions and user testing, so you're basically creating and testing almost in one. You're getting some, I would say that you're not getting as much initial kind of feedback as you would get from doing all of those separately, but what you lose in the potential of getting a ton of input or like a design team to think thoughts, you are potentially gaining in buy-in from your stakeholders. So as Dan said, I think it's a particularly powerful method for showing people, kind of getting people to listen to each other and to have lawyers in the same room as self-representing litigants and for them to understand each other and thinking to how to build a system. So absolutely, and you'll hear more from Dan in just a second as he talks about how he's used this for a particular project. All right, so that is our look at a bunch of techniques. Questions or thoughts about any of those techniques before we dive into some key studies, starting with Ohio Legal Health. You guys are not quite participating in my user feedback on this webinar, but I hope that it's a useful one for you. Fantastic. Rachel, take it away. Do you want to actually control the screen or do you want me to do this one for you? If you could just do them for me. I feel like we work together, you know, you can get. We have a connection there, but okay. Yeah, so as I previously mentioned, I work for the Ohio Legal Assistance Foundation. Yeah, that's perfect. You can, yeah. And we are working on Ohio Legal Health, which is our statewide portal project. We've spent about the past year doing some research stakeholder engagement, gathering data on our users and then design work. And we are starting to actually get into building the website and implementation the summer and on schedule to have the full launch in summer 2019. So I'm going to go over a couple of steps that we took in our user research to kind of get to where we are now. So we started out with a series of surveys. The first survey we did was for providers. And as Laura mentioned earlier on the survey page, when we had a very defined audience of lawyers in Ohio, judges in Ohio, it was much easier to get participant feedback through our partners. We could send out a email through the Ohio State Bar Association, or we could go to the judges conference and have kind of captive audiences and large enough groups of people that it was reasonable to assume that if 10% participated, we would still get usable results. When we moved on to do the voice of consumer survey, the recruitment of participants became much more challenging. So what we ended up with was 800 respondents. About 90% of those we reached through a survey service called Qualtracks. So these were not folks that we were able to go to legal aid clinics and get 800 people to respond. That was a method that we tried and pretty quickly realized was not going to be feasible for the number of responses we were looking for. So that is a cost consideration to take into account when you were looking at that number of respondents. We were able to get some parameters for those responses. However, because we were going with a commercial survey company, it was, again, much harder to get very low income respondents. So we were able to focus in on at least half of the respondents made less than $30,000 a year, which was obviously much higher than the legal aid income limit. So again, as Laura mentioned, not necessarily our ideal target audience. We did get some very useful information from them in terms of clarifying people's main concerns about engaging with an attorney or engaging with the court system as a self-represented litigant. Folks reported that cost and confusion about the legal system were by far their biggest access hurdles to justice. And it also helped us identify a couple of strategies to think about when we were doing our design. So we wanted to make sure that our website is going to be direct and simple. We want to emphasize if there are cost-saving opportunities, we wanna be transparent with people about potential costs of engaging in the court system or hiring an attorney. And we also wanna make sure people feel secured using our website. That was a big takeaway we got from the voice of consumer survey, people indicating unease with giving a whole lot of personal information in order to get self-represented resources. So Laura, if you wanna move on to the next slide. From there, we took kind of a step back to looking at some more national research, which again, Laura mentioned, she did a great literature review and put together a very large diagram that is our mental model that captured about five, I think it was five academic studies and included in the studies were about 250, 300 qualitative interviews. So our final product maps out the thought process of a person with a legal issue through various stages of resolving that issue. That was one great thing about using the different studies. Some of them focused on different elements of that process. So for example, the screenshot that I have here and that Laura was showing you earlier is all about online research. So for that section, we focused it on three papers that really honed in on the online research aspect versus interaction with court staff or some other areas that were also very useful to us, but we were able to really capture the specific findings in each part of the journey, if that makes sense. From here, we use it to develop our user testing strategy. So one of the key findaways from the mental model has to do with making sure that your website is credible because people evaluate online resources very quickly. So we wanted to kind of get people's perceptions on if the current not competitor as Laura said, because we don't compete, but similar resources that already exist are appearing credible as people look at them and see an authoritative resource or if they just see something that is giving them conflicting information, making them more confused. And then a kind of final point on the mental model was that we've also since doing our user testing have been able to share it with a couple of audiences, including our steering committee, which is made up of a lot of attorneys and judges and people who are in the legal world, but maybe don't interact with people who are low income and dealing with legal issues or self-represented litigants on a day-to-day basis. And I think it was very helpful for them to be able to see this representation of the entire process with specific concerns and the specific types of thoughts that people have when they're navigating this as a non-lawyer. Anyone have any questions on anything I've gone through at this point? All right, moving on, thanks, Laura. So for our user testing, we conducted 23 user tests of individuals in five different locations across Ohio. We were very mindful in choosing sites that were in some urban areas, some rural and some suburban locations. We tested four websites that are similar to our desired end product. We estimate that almost all of our test subjects were at 300% of poverty or below. We did not ask them their income, but based on the sites where we were testing, which as I previously mentioned, were community action agencies, which help folks apply for benefits. The highest income that could receive any type of benefit that those agencies assist with was 300% of poverty. So that's how we've kind of made that estimate. Another note is that we only received three male testers out of 23. We are not sure how that demographic skewing happened. And we were happy to obviously get 23 tests. That was a great outcome. But we did make sure to note that, and we were writing up our findings, but they might not be totally representative of our entire target audience. And also just the final note, we were primarily testing to evaluate the navigation and kind of the high level design of the websites. We weren't trying to evaluate content or look and feel necessary for our clients. Look and feel necessarily. We really tried to focus our questions on the specific needs that we currently have. And then as we build, and it is getting past our high level designs, we will go back and do more user testing that will be more specifically focused on things like content or the visual design, things like that. We did not try to do it all at once, which I think is a best practice that you should follow. And now looking at our logistics. We tested in a team of two, my colleague and I, we visited five testing sites and we did that in a work week. I don't necessarily recommend that if you can avoid it. It was a little exhausting, but it can be done if you have to. I think that we've all been there. We tested at sites with walk-in services and high volumes of clients in our target demographic, which is low income Ohioans, who arrived at those locations expecting to wait. Because they were already expecting to have a wait, most folks didn't really seem to be in a rush, at least the folks who agreed to do the user test for us. And that I think was very helpful in terms of recruitment, purposely going to a place where people are expecting to spend time and also acknowledging that they would not lose their place in line for services was also very important. We offered folks $10 gift cards to participate, which seemed to be a good amount. We were at two sites where staff members pre-arranged some appointments, but as Laura mentioned, we did have a fair number of no-shows and the majority of our participants were recruited on the spot by staff. We used two laptops and we recorded our interviews, both audio and then had a screen capture. We had the capability to run two tests at once, but we pretty quickly realized that it was more helpful to have one person kind of drive and lead the test and have another person taking notes. That way, not only could we compare our impressions of the user's experience, but we could also focus on good note-taking and good facilitation and not try to do two things at once. And so just kind of quickly wrapping up with some takeaways. When we did the gift cards, we thought to ourselves, oh, well, we should make sure we have a lot of options. So we have grocery stores and donuts or ice cream.  It turns out that that was a barrier to us. People want to Walmart gift cards. And I think from that, we kind of learned not to be so squeamish about it and also realizing that when we asked some of the staff members at the sites about local grocery stores, they would give us stores that were accessible to them, but were not accessible to their client base based on transportation, especially. So we would be told, you know, oh, we have X grocery store in Cleveland, but X grocery store is actually only in the suburbs of Cleveland and it's totally useless to people who live in inner city Cleveland. As I mentioned earlier, we enlisted staff to help us with recruitment, which was very helpful. I also mentioned this earlier that having a warm opening line was very helpful, thinking about things like where you're from that's maybe not so laden with like a power dynamic is something I would recommend. Each one of us, we had two different scenarios we were testing. So my colleague and I both only tested one scenario. So like I had scenario A, every time I was leading the test. And that was also very helpful when we were trying to document and think about, oh no, I remember this person said this, I know that it must have been on this day because we were in X room and I was running this test, so I know it was the scenario. And that was very helpful as well. As Laura mentioned, record your notes right away because you will forget things. You'll look at your handwritten notes and it won't mean anything. The second to last point I have is watch yourself. So especially when you're thinking about legal issues, and on one hand I think I was a little lucky in that I'm not an attorney. So I did not necessarily feel as much of an impulse to correct folks when they maybe found the wrong information to solve their hypothetical legal problem. My colleague's an attorney and I think it was a little bit harder for her to not kind of step in and correct and interfere with the test a little bit. So just keep that in mind to watch kind of your impulses and the biases that you might have going in. And then finally, it's kind of a self care note to give yourself a break. So we went five days straight and we did 23 user tests, so that averaged about five a day. And we would do those five just right in a row, mostly because we didn't wanna burden our test sites with, oh, well we need people but we don't need people, we need a break. We didn't wanna do that to the staff, but in retrospect, if we had had a little more mindful scheduling we probably would have been a little bit fresher and maybe done a better job for some of those later user tests. Because it is, as we were mentioned, it's not incredibly difficult, but it is a bit draining to be on for a few hours straight. So yeah, that is kind of my experience implementing a user test as a non-user testing expert. Does anyone have any questions before I hand it over to Dan? Just to mention, so both Rachel and Dan's case studies, I've picked not necessarily because they are small, this is not what I would call a small shoot screen user test, which is more like four or five users as I mentioned, but because I think they're super applicable and all of the things will still apply. I'm gonna say to you, Dan. I will turn it over to Dan. Dan, you wanna- Oh, right. I'll just talk for a couple of minutes about paper prototyping, and in particular, I'll show you some examples of truly on a shoestring, very much how much of a shoestring it can be. You can go ahead to the next slide, please. We are drawing these photographs and these examples from a project we have where we're partnered with IELTS Institute for the Advancement of the American Legal System and Stanford's Legal Design Lab on a project called Court Compass, which is a multi-state effort to simplify and streamline uncontested divorce. So far, we've been working in Massachusetts and Iowa. This project is actually a follow-on to the Cases Without Counsel Study that Laura mentioned earlier in the presentation. And so we have been going around, at least in Massachusetts and Iowa, convening a diverse group of stakeholders. Thank you, perfect. For day-long design workshops, this is a good example of a co-design approach where we go from very beginning to the end of the design process in just one day with folks. Paper prototypes are a key aspect of the workshop that we've been doing. And basically, the table teams that we have are able to then test their ideas with other tables by having folks interact with these paper prototypes. So go to the next slide, I'll show you an example of one. This is a mobile app, paper prototype, right? The folks at one table just sort of took some paper and ripped it up into basically handheld phone size stacks of paper. And then they detailed exactly what each screen would have and were able to then hand it over to folks to test and flip through as if they were interacting with a smartphone. And go to the next one, please. And this is an example of a website paper prototype. Again, the same design workshop, but this was done on large format, easel paper, and using stickies and post-its. Again, people would go up to this, the first thing they see, divorce prime, would you like it for us just now? And they flip the page, go on to the next question, go on to the next question. So it can be really, I mean, you can really get a lot of helpful feedback and testing done very, very early in the process with nothing more than a marker and some paper. I think people sometimes if we're creating digital tools, you think, well, shouldn't this, shouldn't the prototype be the, something that you actually a digital aspect of it that they can actually interact with? But no, you can, people are really, really familiar at this point with the basic look and feel of our mobile phones and websites. And it's really easy to just sort of jump into the details of your particular design by just catching it out on paper and having people interact with it. I will say that, you know, as you're doing that, if you're doing it in a design workshop, like we were doing it in Iowa there, and you're working with non-designers as your co-designers, you have to be really, really hard on them not to explain their prototype. You know, the designer stands there and wants to say, no, now you gotta do this, now you gotta do that, and don't forget to do this. No, you gotta keep your mouth zipped and let the other team, the other group of people actually interact with the paper prototype as if they were encountering the digital tool without the designer standing there. And the last thing I wanna mention would be remiss if I didn't, this is from our game represented with statewide legal services at Connecticut. And this was, we've created large format sort of examples of the game, the scenarios that the, or the scenes, if you will, that were available for the game designers to use, and then just they'll cut out to the people. And we use this to design the, or to come up with some scenarios for represent renter, which is the second of the two games. We brought these to our co-design sessions down in Connecticut and up in Maine, and had people play with these as they thought about what the characters might do in the game and thought about how the characters would interact with each other. It was more of a play tool in a way, really, than a paper prototype per se, because it's not a testing function, more of a design process function. And it's very playful and kind of fun. And so we've got those here in the lab and we use them whenever we are sitting down and beginning the process of thinking about using that platform for another purpose. And I just encourage people to be as creative and fun and quick and down and dirty as you can with paper prototyping, because you can just hammer out those ideas, get some immediate, very quick feedback, and then pivot to wherever you wanna go. Awesome, thank you so much, Dan. Yeah, I would echo paper prototyping is a fantastic technique that not enough people use, I think. And thank you, thanks to Dan's, particularly Rural Wintour, through his own case study, for example. Let me give you a note of the time there. I appreciate that, I apologize for pushing you to the end. We'd love any questions you have. So you've got three people with a fair amount of expertise here to take on kind of questions you have or thoughts you have. What questions do you have? Or, sorry, do you have anything to ask the group? Ask Dan and Rachel a question. So maybe I would say, so for those folks who are thinking about getting started, so they haven't done a lot and we've just presented like 19 different things that they could do, where would you encourage them to potentially start? Can we narrow down to just a finite number of things that we think might be a good place to start experimenting? Oh, boy, that's a tough question for me to answer without. I mean, again, I hate to say it sound like a classic lawyer, but it always depends on the problem you're trying to solve. But I think research is actually a really helpful way to get started. There's a lot of study, you know, academics. I'm here on a university campus. I can tell you academics like to write papers. And they spend a fair amount of time thinking about issues, thinking through issues. That's a great way to start to frame out the problem that you're hoping to address. And it can be a place where you will start to identify, okay, well, this has been done before. That has not been done before. Here's the opportunity pass. And that's, you know, academic research is a good place to start. And no one's probably going to tell you to stop. You know, no one's going to say, no, don't do that. Don't go read academic papers, so. Well, you don't want to go too far, right? You know what, you're not doing a lot of your articles. You don't want to research all the way down to the Magna Carta back at MBE. You can get a good environmental scan of what the state of the play is, the state of art. Yeah, and I might, if I'm answering my own question, I might say, go talk to anybody in your target audience about anything. So plan out what you're going to ask them and go someplace where people are and sit down and talk to four people about their own experiences. Just because I feel like meeting the actual people that you're serving can have a really powerful impact almost beyond what you are actually creating. Yeah, I would echo that, Laura. As someone who works in an office that doesn't provide direct services, that was an incredibly important part of my user testing experience. Just the few brief kind of introductory questions we did before we got into using the sites was very, not only, I think, technically full, but also kind of grounded me in who I'm building this for. And I think that that's kind of an invaluable experience. Yeah. Fantastic. All right, well, let's... Fantastic. Okay. So, I mean, how often is basically as often, if so, you ideally have been testing incrementally as you build it and then every time you do a change to it, you would test it.  I think realistically, it's a really hard question because there's so little that's been done. What I would like to say is, realistically, you don't need to test every little thing because you can assume that your site is going to work like other legal aid sites or other legal help sites for most things, but that doesn't help you because nobody has tested anything or so few people have tested anything. So, I would start with, I mean, do as much as you can. So, I am in charge of research at Florida and the Florida Justice Technology Center and we just set a goal for ourselves to do two serious outreach studies for per year. So, and I think that that is, and that's about 20 user tests per, so you can also split that up. So, that'd be about 40 user tests per year is what we've just set ourselves as a stretch goal in Florida. So, we could also do a full session on active A-B testing, putting together design processes so that you are testing live iteratively as you roll out new resources. Yeah, that's a great point as well. So, basically, we can stand on this. It doesn't have to be a separate thing. We're better than that, yeah. What are the questions we have? I mean, one of these here is what would be kind of your recommendations on most relevant few articles from the last two or three years. If somebody was going to go read a article on this, what would you recommend?  Because you just suggested, hey, start with the literature review. Look at the research. What article would you recommend? Well, I will take the easy path and say, so I've just posted to LSN TAP the mental model that Rachel described. And that summarized five different national studies, well, four national studies and one Canadian study. And that's, I think, where I would recommend starting. Where all five of them are studies of kind of how self-represented litigants think about information to help themselves. That was posted to our email list. We'll grab that and put it into the blog summary of this particular webinar as we post the video. Yeah, great. Thanks, or do we need to wrap up right on the hour? So, there was a question, which is the lit review from Ohio, and I assume that is what part of that was. Yes. Okay, excellent. Perfect. Yep. Yeah, so that's posted. We also just recently posted and can repost. I can send around again the full report from the user research that Rachel mentioned, which among other things had, I mean, it has the results but it also has the entire methodology. It has exactly what we did in the entire guide for it in it. So there was an interesting question about how can government agencies use gift card ideas for participation, it seems like, and not doing so because of using public funds. My quick answer on that is that compensating people for participation in an activity, I'm not familiar with any restriction there. The restrictions that I'm aware of come into things like competitively bidding contracts at a certain size, that type of stuff, but a nominal compensation is extremely common in the legal services industry for people who are giving their time to really improve these resources. And I feel like Rachel, you guys thought a little about that, yeah, if you know any answers to that question. So we didn't really get into it. We did have a testing site at a court self-health center and our office, which is a non-governmental organization was offering the gift card within the context of the court. And the court didn't seem to have an issue with that, but I think it probably may depend on your individual administration. Got to make that. What else do we have, Brian? I think that covers everything. We've had a very interactive session today. We're gonna have that summary up. And then the entire webinar will be available on our YouTube channel. Thank you so much for putting this together. Today, I really appreciated. Thank you both to Dan Jackson and to Rachel Harris and then to Laura Quinn for putting this together. It's been a wonderful session. And thanks everybody for coming. Thank you to Dan and Rachel and I hope to, and I'll see you at another seminar soon.


Dyma'r ddweud. I'm Helen Claire from JSC, the UK's National Research and Education Network. I'm normally home based, so this is life as usual for me, up in sunny rural Northumberland, in the UK. I'm keeping my camera off because I do have network problems here normally. I just did a little bit of an introduction to myself before introducing the project in a session, but I am focusing on open science and open learning in that context and the European Open Science Cloud. So, while I'm doing my short intro, I'd appreciate if others might give a little bit of background about whether you have any knowledge or experience in the open science, open access area. In terms of my background, I do have a background in training for a number of years in learning technologies and libraries in particular. I worked as a JSC service called Net Skills for a long time, so if you remember that, and I was around for the early days of VLEs, training lots of institutions and the basics of their platforms and good e-learning practice. I've also attended OER 11 and OER 15. But the last five years of my career, I've been very much focused on scholarly communications and open access. It's interesting to follow a presentation on policy because that's been my life for the last five years. Helping institutions comply to open access policies and the complexities there. But in the last six months, I've been working on a European funded project called EOS Synergy, and this brings me back into the world of online learning again. So, that's reason I'm here. We're in early days of the project and I'd love to get feedback from you all. Because from what I'm seeing, that there isn't too much of an overlap between the communities, the open science communities and the open education communities. Please correct me. I do see a handful of familiar names in the chat panel. So, it would be good to hear from you too. So, I'm just going to do a bit of a background to open science and then get into the project, what it's doing and what I want from you in this session. So, in terms of open science, lots of different definitions around open science itself is quite a controversial term because it's not inclusive. A lot of people feel it's not inclusive in terms of discipline. It's not just about science. So, in the UK, open research is preferred. Not all definitions, such as the one on the slide that I've included there, include OER. Many are focused on the research lifecycle. I guess the key point is that it's about opening up the whole research process. So, opening up the ideas generation, the peer review, the actual working data and then the outputs at the end. So, in terms of the European Open Science Cloud, this is a very, very ambitious initiative which started around 2015, which is enabling the opening up of the whole research process as it says on the slide. You can see various different aspects of this and motivations for doing it, but the idea is enabling the research data to be opened up. I should say that it is quite broad in its interpretation of open science, as I said earlier, and broad in its interpretation of what we mean by data. So, it can be all sorts of different outputs including publications and multimedia. My first introduction to the word, to the European Open Science Cloud, I was told that it's not just European, it's not just open, it's not science, and it's not just a cloud. So, you sort of hear explanations of those various aspects as we go through the presentation. But it is an ambitious initiative and at its core of all the activities that are at its values, which you can see outlined on the slide there, and again, I think a lot of those will be very familiar to this community. Some that might not be as familiar are the idea of fair or close where necessary. So, we very much talk about fair in the open science community, so it's findable, accessible, interoperable and reusable. So, some data cannot be open because there are legal reasons and ethical reasons why it can't. And also another term there, it may not be so familiar around long-tail research. So, it's not just about big data that produce a lot of data that may be well-structured, but most researchers produce some kind of data in some format, and that's the small data. So, in terms of this, what EOSCA is an ambitious project in lots of different ways in terms of the technical infrastructure, but the part that I'm most interested in is on the skills side of things. So, there are challenges around skills and around culture. This report, in the early days of EOSCA, identified 10 challenges with three of them around the skills area. So, there's a shortage of data experts, there are barriers to developing skills, a lot of those may be around the reward structure. So, it's focused on articles and not necessarily data publishing and reuse, and there's a lack of intermediaries to support researchers. So, there have been a lot of initiatives over the years and continue to be initiatives focused on skills. So, I'm just going to explain a little bit about EOSCA Synergy, which is the project I'm working on now. It started back in September, so we were only six months into it, and it's defined as one of EOSCA's regional projects, i.e. it's about integrating the various data and services provided in different nations into EOSCA, and then also integrating EOSCA back into the national infrastructures within those nations. So, it is about embedding and it's about sustainability. Synergy has a wide range of different activities, and some of them, like I say, are about bringing data and services on board. So, we have a whole range of interesting topics, such as entomology, ethnoarchaeology, geological data, data from the Latin American giant observatory, and then services layered on top of those in terms of things like earth sciences, climate genetics, and you can see the news story from just this week around some work we're doing to open up research and provide services to analyse those research around COVID. So, in terms of what we're doing, I'm in work package six, which is around training and skills for EOSCA, and we're there to support integrating, as I say, integrating training into services, where they're trying to improve usage of EOSCA and develop EOSCA-related skills, so those can be around implementation of FAIR, or research data management, or a wide range of different skills that have been defined. We're also looking to develop a learning platform, an open learning platform, if one is needed, for those that don't have one, and we do have a particular focus on online delivery, so we're not looking at face-to-face delivery models, we're looking at online delivery, and we're supporting a train-the-trainer model again, because we want to be sustainable, we want to support those who are creating content in how to create content to support EOSCA and related skills. And then finally, our final element is around embedding this into national institutions and national infrastructures. So, that's the background. What have I done so far on my particular task? I'm leading a task around developing methodologies, so how do we create good practice in developing content, and how do we support others in creating that? What I've been doing particularly so far is talking to an awful lot of people. There are a lot of initiatives, so many logos on the slide there, from the Carpentries, Open Science MOOC, Digital Curation Centre, FOSTA, a lot of initiatives in the Open Science training area. So, we've reviewed those to make sure that we build what's there and don't duplicate. We've also reviewed open learning platforms and looked at current online learning trends, partly because I've been a little bit out of this for a while, so I needed to refresh my memory on a lot of areas. And then we've done some initial selection of platforms based on the EOSCA values. So, in terms of what we've found, a lot of this slide will be very familiar to this audience, I think, to make sure that the project team were coming from a common understanding of the trends in online learning, and particularly around the word MOOC, because in our project proposal, we had the word MOOC mentioned an awful lot, and I wasn't convinced that that's actually what we were planning to produce, what we are. That would be one of the range of approaches that we will support, but we're not just talking about MOOCs here. But key points for us really is that we've moved away from a single monolithic platform as I used to train in the early days of VLEs. We're now talking about a suite of tools used for specific purposes and bringing them together, so the interoperability and the independent standards in terms of our observations from our reviews of those many initiatives, we found that there were common approaches within each of them. Most of these initiatives took a train-the-trainer approach, but this was mainly for face-to-face delivery. So, a lot of our boot camps and workshop sessions. Yes, I do see that. Lleba Gemma is definitely a partner that we're working with. I've just seen that in the chat. Many sites had collections of training materials, so attempting to have catalogs. When you looked into them, these were usually power points or resources from face-to-face workshops. Finally, there were training communities, so the greats of communities developing in different areas. In terms of online delivery, we found that there's not much focus on structured online delivery, so content structured in the way that you would in a MOOC, it tended to be around, as I said earlier, sharing power points, workshop resources. There were some webinars, however, in terms of platforms. Moodle was a dominant platform used by these projects and initiatives, and in terms of the features used within those platforms, it's tended to be a fairly standard set, in basic assessment, certification, media and communication channels. But talking to some of the projects, it was quite interesting how the different tools were used. For example, the Open Science MOOC, they tended to use, the LMS was really just a presentation layer. They developed their content in GitHub, they talked in Slack, they shared videos on the Internet Archive, so it was really just using presentation and for some basic assessment. So what did all this mean for Synergy? Well, our training approach for online learning particularly we felt was still needed. We felt that many of the initiatives were tied to a subject community and that we could fill a gap for those that weren't part of a community. And in our review of platforms we came up with a short list of a selection that matched our EOS values and I believe they do match some of what's been mentioned in the chat there. In terms of the project initiatives, foster and up to you, and the latter in particular, you mentioned along the lines of what we're thinking and very much matches EOS's approach of sort of federating platforms. So I'd recommend if you haven't come across up to you taking a look at what they do and it also turned out that they happened to be based out of the Poznan Computing Centre which is where our work package leader is based. We did also find that there was still a need for an open learning platform. The Open Science MOOC during the lifetime of this project found that their platform had actually gone bust the platform that they were relying on so they were looking at a new host. So we've been focusing on online learning and this is where we plan to go. We plan to do more user needs analysis, develop use cases, personas. We plan to develop the content of the train of programme. We plan to look beyond the Open Science initiatives into other relevant contexts so in terms of online delivery so computing, data science, teaching. We plan to look at that platform so a very agile approach looking at the tool stack that we wanted and the standards and metadata that needed to underpin that. So that's what we planned to be doing and recent developments have somewhat overtaken us. Everyone is now online. Everyone is sharing tips and good practice. So, hi. Sorry, just to let you know you're now in the final. You've got a few minutes left so it's very much just the heads up. Okay, thank you. So those initiatives that we saw previously that hadn't been doing a lot to support teaching online have now started to do it. So, for example, the Carpentries. Everyone is sharing tips and good practice and it seems everyone who didn't have a platform before now has some kind of platform. For example, again, back to up to you they're now supporting the whole of the Polish school system and they are opening up their platform and providing this open up to you platform. So what should we do now? Well, we'll continue with the agile approach for platforms and we'll carry on focusing on that interoperability and reuse of content. In terms of what I plan to do I didn't feel that this was a good time to start sharing more tips because there's so much out there and we're bordering on overload. So what I thought I could do is actually just start talking to people start gathering stories particularly from that open science context and learning from that and it was interesting in the introduction yesterday that Francis Bell highlighted this at this point exactly we should just tell our stories and work out the significance later and we're not the only one doing that we were actually approached last week by an organisation called 411 who are a global school locations organisation to coordinate a global initiative to do this. Capture these online learning stories in the open science context and we can also look beyond that emergency delivery obviously the longer term impacts of the change and what they mean for the open science community. So we're at the end really and what I wanted to know really from you is to know what you thought of everything that we've been doing again trying to join up the community there's a lot of expertise in the open education community that we can learn from where can we add value do we still need to open learning platforms is that something that will be useful what can we learn from the OER community and I saw a quote from Donna Lanclard this morning which I thought was really interesting is that it's a kind of assumption that we we're all going to go online and that's that but actually you know it may not be so straightforward there may be more consequences of this than we currently realise. So can you hear your comments, suggestions, questions either in the chat or please get in touch afterwards and you're only listening to the recording the same goes for you too. Brilliant, thank you ever so much Helen. Do we have any quick questions for Helen before we round off this session just having a look at the chat I can't see any explicit questions lots of useful interaction and comments there for you Helen that you might like to kind of have a look through within the recording. Juliana's got her her hand up are you happy for me to hand over a mic quickly just for we do really have one minute. Excellent excellent presentation very much needed this review of platforms and aggregation and also showing I was engaged in the open science movement and it's quite original the approach and one thing is I work on the extraction of 1500 1500 cases and almost two millions of cases of a fixture and research gate about the quality and the practices of open data and there is there are very low skills and awareness in the research community so I think we'll see the impact of your project in the long way and we need very much to analyse that impact because by now we have the tools we have the policy context but the researchers this extraction was done recently last year and the practices are not at bouncing yet so how do you see the possibility of analysing the impact and engagement and practices further okay thank you Is this in terms of the current context or in general? Relating for example open data because it's one of the most important practices within the context of open science but other for example crowd science, responsible research and innovation are you planning to analyse the impact of training over researchers skills because as far as it is my understanding these are very low Yes I think part of this project I've become involved in some European wide training communities and I think evaluating the training is a key part I think yes we want to do it but I think it's back to those difficulties sometimes of capturing that data but I'll certainly take that question back to the communities that I work with and ask what they're planning to do We obviously need to look at that in terms of our project and how we will assess the impact of our particular training Brilliant thank you so much Helen Thank you for the question Juliana I'm going to have to close the session now because we need to set up for our next session in this room which we starting at 12.15 I just want to


We can't predict when an earthquake will hit. To minimize personal impacts, Californians should proactively prepare and never rely on luck. Californians have the My Shake app. They can give them early warning about an impending earthquake and a few seconds to take protective action before the ground starts to shake.  Prepare ahead of time by retrofitting your home to strengthen your house's foundation. When it comes to surviving earthquakes, it's not about luck. It's about being prepared and having a plan. For more information on how you can prepare for an earthquake and other disasters, visit news.caloes.ca.gov. For everyone at CalOES, I'm Brittany Peterson.


From you today, are you among the the creatives that will be maybe making a speech talking to the government? No, I'm not one of the creatives making a speech if I get the opportunity though by me I will have something to say but for sure all I hope is that we get the support that we need because we've been You know doing this by ourselves for a very long time. Yes, they are recognizing us now But I just hope they recognize the need to help us because the struggle has been real. Will this work? I hope it does. I mean you can only cast a net in the ocean and hope it catches a fish or two That's what is happening here. This is a very wide net that is being cast Do you think like now the creative now the industry the creative economy will move to another level reach out to another level? I hope so. I mean we can only hope at this point. It's nothing has been done Now is when we are starting to do things So we can only hope that this is going to you know cause a ripple effect of something else because we know we know very Many non starters as far as trying to quote-unquote help artists is concerned I remember during COVID we had the 100 million that was supposed to go to artists So we just hope that the effect will be will be felt and you know have a long-lasting effect All right. A message to creative regarding Talantahalab Let's cross our fingers and hope for the best that I honestly don't know what they because you see they're launching it today So we don't know what exactly they have in store for us. Yes, all right. Thank you so much for the time I wish to catch up with you later. Yes all the best All right


Do you have any moments in your life where you look back and your skin starts to crawl from embarrassment? Personally, yep, I have. I would say I probably have 10 or 12 recurring ones for a long time really bothered me and It's so great that I get to talk to people in my work But if you are One of those people that has many of those cringe memories that come back Just realize you're not alone We all have these and I want to give you a little bit of a feedback in terms of how you process these memories Because they're so intrusive. They're the you could be in the shower one morning And just this memory pops up into your mind of one of the cringe moments that you felt deeply embarrassed and awkward in Maybe you have a lot of regrets and a lot of shame tied up in it Now what's driving these memories is a fundamental core belief It's one of the three core beliefs I talk about And it is the belief that they're in being defective in that there's something wrong with me shame guilt Are tied up in this now In terms of processing this One exercise. It's not an easy exercise to do is just One day in journaling what I invite you to do here is to sit down and Write down, you know, you can call this journal entry cringe moments or something right where you look back in your life and you look at in sequence the Moments in which you felt that really skin-crawling embarrassment Usually, of course, it's around other people The bigger the group of sometimes the more intense it can be Just to write them down and to see we're going to do like a stock take here and evaluation of How many of these moments are there for me? and How often do these thoughts intrude on my life because They pop up seemingly random, but For many of us they happen so often that it can kind of become an issue, right and it can really start to erode away at our sense of self-worth now the secret to this if there is a secret is How how do we let this go? How do we process it? And I know many people have approached this differently I know many people have Tried to like reframe memories or change memories even which to me seems a Little bit difficult to do. I don't think it's about that. I think it's it's about looking at the memories and Here's the thing. Here's a secret You have to give yourself complete permission to Experience moments like this What I'm really saying is It is how you are perceiving these moments in your life and using them against yourself That is the main problem Give yourself complete permission To experience moments like this in your life. You have to give yourself permission You now I know what does that mean give yourself permission You have to come to terms with the fact that you're allowed to do that That thing that made you your skin crawl Okay, and it's a reason I'm talking about this is it's like we don't often talk about this I don't hear many people talking about this issue And it's it's one of those things that can know away in the background with us sometimes So for me personally many times like public speaking I've given loads of talks over the years Many of them most of them almost all of them have been good few of them have been cringe-worthy a few of them I have really made a like anxiety talk over or something and I Found it difficult in those moments and how I was processing those experiences was Dysfunctional right so that needed to be worked on and ultimately when you look at it Give yourself full permission allow yourself to do it It is how you are viewing these memories That is all important and if you realize if you give yourself complete permission Really to even celebrate the fact that that's what I did that happened to me Okay, I Can guarantee you that this won't matter so much, but I'm going to say it anyway. No one else is thinking about these moments not one soul right and But if they're popping up in your awareness again and again Really the issue is okay. I need to learn to let this go.  Maybe you had an anxiety attack or Something like that said something wrong Maybe you were asking something out on a date. Maybe You said something wrong in it in there like a job interview or something like that Giving yourself permission to have these experiences Will mean that you're going to free yourself up to be way more creative and Open to risk in the future But really the most important thing is that these memories that pop up and they're like a thorn in your side sometimes They will start to fade off. In fact, here's another thing I would invite you to try every time a memory like that pops up into your awareness Amplify the memory Really go into the memory proactively because usually what happens is when these memories pop up We want to get away from it, right? what we need to start doing proactively is when these memories come up amplify it welcome up that memory and celebrate that moment of your life See it as one of the Best moments in your life That's what I mean by embracing it this issue is so common for us, but We don't talk about it that often and I think That tells us that we're trying to escape these moments, right? We don't want to look at them Many many people usually successful people Component people have had these experiences. I've seen a lot of them online of People kind of have an awkward moment, so whatever So you're not alone in this, but you need to learn to celebrate these moments Really give yourself permission to be a person that's allowed to do that I'll leave it there today short video, but I hope that was useful. Take care