I want to thank you guys so much for coming in today and talking about the redesign that you guys did of a law help Minnesota. I'm turning it over to you, Dan. All right, thank you very much. I, as Sartre introduced, we're going to be talking today about the project that we completed over the past year, rebuildinglawhelpmn.org and its presentation given by my agency, Electric Citizen, as well as a member of the Minnesota Legal Services Coalition. Just a quick introduction for today's presenters. On the left there, that's myself, Dan Moriarty. I am the CEO and Creative Director at Electric Citizen. I am joined today by Tim Broker, the Technical Director at Electric Citizen. There's Tim. And I'm also joined by Jennifer, aka Jenny Singleton, the Program Manager at Minnesota Legal Services. Hi, Jenny. Hi, everyone. Excellent. So now you know about us. For what we'll be covering today, sort of a recap of what was already shared, we're going to be talking about some of the biggest components that were part of this project, including a legal resources tool, a legal triage, online application, legal service provider tools, multilingual. We're going to touch on the different phases of the project and sort of how each one impacted the work we were doing. We'll touch on the user testing that we completed as part of the project. And we'll talk about the relationship between our agency, aka the vendor and the client, and some of the challenges that we face together and how we overcame them. And lastly, we're going to talk about how we manage the big rocks, which I will sort of explain that concept as we go here. So again, about the agency, Electric Citizen, we are a Minneapolis-based agency. We've been around since 2012. Prior to that, most of our staff, including myself and Tim, have been working on the web since the mid to late 90s. So we've been doing this a long time. We focus as an agency, we tend to focus on the civic sector, which we define as government, higher ed, nonprofits, arts, science, research, and the like. So the law help project that we're going to talk about is a perfect fit for our mission. We are open source advocates, and particularly with the content management system, Drupal. We're a team of highly experienced Drupal experts. And if you're not familiar with Drupal, you can certainly Google it, go to Drupal.org and start to get familiar with a really modern, well-supported, and well- used content management system and framework. And lastly, if you want to learn more about us, we're at electricscitizen.com. And now I'll turn it over to Jenny to tell you about her agency. Yeah, so I work, again, with Legal Services State Support, where our project of the Minnesota Legal Services Coalition, which is just seven legal aid organizations in Minnesota that have gotten together. And my office's role is to increase coordination amongst all of those organizations. So we're, along with those legal aid organizations, trying to improve access to justice for all Minnesotans. And we do that by increasing coordination and really leveraging technology. So some of the projects that we manage that you see up here are LawHelpMN.org, which of course is what we'll be talking about today. And then we also have our Education for Justice program, which creates a lot of the content that you'll see on LawHelp. And in addition to those programs, we have an online legal advice program called mnlegaladvice.org, and then a website for the legal aid and pro bono attorneys in Minnesota. So for this project, again, we were really focused on our statewide legal information website and trying to make sure that that was leveraging the resources that we have on the website and making it as easy to navigate that site. And there were three main things that came together in 2017 that I wanted to talk about to provide a bit more background about the project before I turn it back over to Dan and Tim to walk through what we actually did during the redesign process. So prior to our current website, we still had a statewide legal information website, LawHelpMN.org, same URL, and it had a lot of really good information. But again, in 2017, there were three different things that really coalesced and came together to make us see the need to create a redesigned website and add some additional functionality to that site. So the first was some strategic planning that we did through a Justice for All grant through the National Center for State Courts. And we had three lead organizations, the Minnesota Legal Services Coalition, our state judicial branch, and our state bar association along with a broad swath of other stakeholders come together and think about as a system, what can we do to increase access to justice along a continuum of services. So everything from a fact sheet to static forms to automated forms to clinics and full representation including also places where people can go to get assistance like self-help centers and law libraries. So we wanted to create a website that would make it easy for users to find all of those kinds of resources and not be overwhelmed by them and find what resource would be most particular how they were situated. So one of the goals from the strategic planning was to create a triage system on the law help site, which we'll talk about later, but just in a very brief nutshell, a triage system that would refine the user's legal issue and then gather some eligibility information and based on that user data provide self-help information and referrals to legal organizations that would help the user navigate whatever legal situation they were faced with. So in addition to that justice for all strategic planning, in 2017 our state court also commissioned a study on the legal aid intake infrastructure in Minnesota, so focusing on civil legal aid and tracking every call that came in, every contact that all of the legal aid organizations in Minnesota made with clients or potential clients. And one of the many findings from that study was that clients often would call organization A saying, I have a legal problem about this and that organization would say, oh, you really should call organization B and organization B would say, oh, we also can't help with that. So try calling organization C. So we called that phenomenon Bounce and one of our goals for potential clients had to call and make it the entire system as a whole more coordinated and easier for users to navigate. And to do that, we needed to have up-to-date and really detailed information from all of the organizations providing legal services in Minnesota. So we'll also go into how we tackled that problem later on in this presentation. But one of the key goals of this project was to create a way that organizations could easily keep really detailed information about their services up-to-date so that legal aid organizations in Minnesota know what each other are doing, what types of cases they're taking, what eligibility is attached to those cases, and that that information can also flow out to the public so that the public has a place they can go to find what's the best organization for me to call for this particular problem that I have. So then the final thing that happened in 2017 was LSE released its statewide website assessment. So every state got an individualized scorecard with information about their statewide legal information website strengths and weaknesses. So at state support, we sat down with Minnesota's results and looked at what can we do to improve our website and capitalize on the strengths that we have to make the site easier to navigate and make it easier for people to find the information that they're looking for. So we looked at our existing website which again had great information and realized that we really had three primary channels of information. We had a self-help library under the Understand Your Legal Issue section, a provider's directory and Refined Legal Help, and a link out to an online application system where somebody could sell out information and shoot it over electronically to the right, hopefully, legal organization. But each of these parts of the website was very siloed and if say you had an eviction problem, you might find a fact sheet about evictions and what your rights are under the Understand Your Legal Issue section and then you have to go to a separate part of the website to find what legal aid organization might be able to help you. And then if you wanted to send in an online intake application, you would have to actually go to an entirely different website, enter in all of your information and then send that through to the organization. So our goal with the redesign was to bring all of the great resources that we had on Law Help Together as a cohesive whole to make it really easy for users to find all of the information and all of the resources that might be able to help them with their given problem. So we did that through working on our design, working on some content curation, and then again getting better information about what the organizations were doing and tying all of that information together with our triage system, which we call the Law Help and then Guide. And Guide users through all of the information on the website out with an interactive tool to get them both the self-help information and the legal referrals that are most appropriate for their particular situation. So again, we've been really pleased with the results. I've got screenshots up here of the prior website and the redesign website. The redesign is on the right and we've got a really clean design that we love with a really clear call to action with that big Start Here button. So if people come in via the home page, we're hoping that they'll just click that Start Here button and then that will take them to our triage system where they can find that really curated result for their particular situation. Similarly, the self-help library has a really clean design. We love the icons that Electric Citizen helped us develop and helped the design that they helped us come up with. And it just makes it a lot easier and less overwhelming to figure out where you need to go within the website. And of course, everything is mobile optimized. So we had a mobile first design approach where the site looks great on a phone. It looks great on your desktop. So we've been really pleased with the results and I'm excited to hear Dan and Tim share a little bit more about their side of the development process and how that worked.  All right. Just wait for this button to show up. There we go. Okay, so you should all be seeing my screen now. A quick note about rocks. Everybody knows what rocks are, of course, but for this particular project, as I was reviewing sort of our meeting notes and thinking of where to organize this webinar content for you, I came across a meeting where we sort of identified the big rocks for the project. So it's not language we always use, but we used it here and I think it's effective. So if you can imagine any site of this scale and there's tons of little details to keep in mind, lots of little rocks to pay attention to. But like a lot of projects of this size, there's a few really large standout items that need our biggest focus and attention and consequently, we're the biggest deliverables as part of the site. And so what we're going to do today is go through each of the biggest rocks that we identified and worked through for this project and take it from there. So as you can see here, when we went through our initial planning meetings talking about what we needed to deliver for the client, we identified this list of biggest rocks or biggest deliverables that we needed to focus in on. So number one was a resource library, all the sort of different content of forms and links and such that we're offering to people needing legal help. The triage tool, which we'll get into here shortly, guiding people to legal help. The provider tools, a set of tools for the legal providers in the state of Minnesota to find referrals and to manage their own content online, multilingual, serving the site up in different languages. And then those were four, but then there was also not to not last or least was to refresh the overall design and user experience of the site. Make sure we take care of security and privacy, talking about intake process. Once someone has found legal help, how do they find a source for their legal problem? How could they get in touch with the legal provider? SMS integration, ability to send content from the site via text. And lastly, how to manage user feedback. So those are a whole bunch of things we're going to focus in on each of those in a section. So we're going to start off the rock conversation with rock number one, which was the resource library. And when it comes to any of these deliverables, these rocks that we're talking about one of the first questions we are asking ourselves as a team and worth the client is what's available now? What problems are we trying to solve? So when it came to resource planning, resource library planning, well, the very term resource library, I don't know that it's actually used on the site in that way. We ended up phrasing it as a self help library. But the idea internally was how can we organize these different resources that the current law help site offered into us a format that was easier for people to digest and find what they needed. So you can see a screenshot of an example of what that would have been like before, where you would browse by a certain topic in this screenshot example, we're looking at permanent resident status. And you get a list of a running list on the page of different resources from things like fact sheets to links to other websites or forms on the left hand column. There's links to things like booklets or legal forms. And so there's a whole bunch of things that are fall within this whole resource library planning. And so our job was to make this whole process easier for people to quickly scan and find information on a variety of devices for mobile to desktop to try and reduce some complexity if we can to come up with a solution for people searching for these and just overall a visual refresh. And so what that meant. So what we ended up with as Jenny had showed earlier was something like this. So on the left hand side, this is a new browse self help page where this also can be accessed below the fold on the homepage, a series of categories with new icons and some revised terminology simplifying the categories that people might search for resources by. And then on the right hand side, you see the end result of how we're laying out these pages. So we went from from something like this to this where we're segmenting this out. And I'll touch more on the design part of this as we go. But some of the top goals that we considered when when redesigning this was the users may be facing emergency and they need to quickly and efficiently find information. They're typically starting with a primary topic and within each topic is a primary resource as opposed to choose from a dozen different ones. So in this case fact sheets became the primary resource that we're going to highlight in each topic. And we wanted to use labels that were easy for users to understand. So one of the things that we did was we had to go into the current site and audit all the types of existing resources, identify what each of them do and then look at ways that we can pair them down or combine categories are the terminology that we're using. Does it make sense to people? It might make sense to someone the legal background. But does it make sense to someone who doesn't have that kind of background? Is there any kind of adjustments to the language we're using? And what is the user experience? And so as you can see in this column of the right, I don't know that this is 100% accurate, but this was an initial attempt at us auditing the existing content identifying over 100 different fact sheets. There's booklets. There's so over 700 links going to other sites. These are all sort of resources that were available that we're trying to reorganize and make easier for users. So once we've identified these categories, if you will, or the types of resources and adjusted the removing some that we didn't need or combining some categories, we go to wire framing the new pages. And if you're not familiar with wire framing, I'll get to that in a moment. But basically what we're trying to do is simplify the number of choices, right? So if I were to look at the old site, I wanted to find resources on that homepage, I might look at each of those highlighted areas where different types of resources were were being called out. So there's section for fact sheets. There's several links in the footer. There's something in the sidebar. Let's make this simpler. So when we wireframe, we're looking at organizing the content on a page, we're just finding the priorities for each page, the hierarchy. We're not looking at the style so much, but we're just looking at how can we organize this in an easier to use format. So as you can see, what we've done here is in the wireframe, we identified, we're going to make fact sheets the one resource type that is highlighted first. Right. And so as part of simplifying the page, let's get rid of the different side columns that and just leading people down the page from a series of content that we've prioritized for them. If the fact sheets aren't what they're looking for, they scroll down and there'll be other resources such as forms or links. We'll add a search bar to this page so people can search for particular types of resources. At least that was the concept in this wireframe. Another concept was let's reduce some of the visual clutter by making use of accordions. And so if you look under fact sheets, you'll see four results with a with a button to view more. And that's a concept that we did take through design and into production and ideas to give people a sense. Okay, there's this content for fact sheets. There's also other content on the page. If you want to see any additional fact sheets, you can reveal click and reveal to see them all but to not make people scroll too much to see what else is on the page. And you can also see from the side that we're looking at this when we wireframe not only from a desktop but from a mobile standpoint as well. So those are accordions, those areas where you kind of hide that content. How does that work on mobile and have how responsive is that? Sure. So well, for mobile, it can work the same way. It's just a matter of touching a link to reveal additional options. What we typically do is one of the ways we're thinking in terms of mobile first is is by eliminating some of those sidebar content. You can see from the big image on the left to the smaller one on the right how content just realigns itself and stacks up from a mobile standpoint. But to the accordions are still triggered for mobile users as well, they just get the same sort of experience. It's also something that we identified as a way to make them accessible. So someone, for example, not using a mouse can tab with their keyboard to these different accordions and use their keyboard to expand and collapse them. So we've got two more questions here. First, like what could you explain a little more like what is a fact sheet or what you decided to put into those? Sure, actually I'll let Jenny answer that one. Yeah, I can take that one. So fact sheets were a content type that we had prior to this redesign, and they're basically two to five page plain language explanations of common legal topics that folks are facing. So like our most popular one is explaining determination of parental rights process and what that involves and what your what your rights are relative to that. So we have them both in PDF because our partner legal aid organizations actually hand physical copies out to clients really frequently. And then we also have the online version of them where they're accessible. Next question was what what type of user testing have you guys done with accordions? We've got somebody from another legal service or another org that had had a rough experience with that. So Oh, so I can say that later in the presentation, I do touch briefly on our user testing that we did, including people looking for content. Overall, on the site, we ran a series of tests for them to do both on mobile and desktop devices. And in our observations in that user testing, which granted isn't coverage of everyone that's using the site, it's just a small segment, but we didn't have an issue with that. And I'd be curious if Jenny or her team had heard any additional feedback since then about any issues with that. Yeah, we haven't actually had any feedback with people having problems with the accordions. So as far as we know, they've been working well for people. Part of the part of the test suite was find such and such and that such and such was buried within an accordion. Go ahead, Jenny. Go ahead. So the testing would be, for example, find information about how to get your renters tax refund. And then people would try to find that fact sheet and then look at the fact sheet. So we were asking them to find, I don't think, specific details within the fact sheet itself. Okay. This is Tim. I'm going to jump in too. And Jenny, you can correct me if I'm wrong, but I think we also have a assorting mechanism here that will show the most relevant ones based on any given category. And then as they get less relevant, that's when they sort of disappear into the accordions. I recall you have something like that working. Yeah, so we have accordions in a couple of different places. So we are able to rank each resource. So if replacing your permanent resident card is the most important resource for our fact sheet within a subtopic, then we can have that rise to the top. And so anything below the four showing resources would be less relevant. Don't recall if any of our user testing had people go into that particular type of accordion, but then within the fact sheets themselves, we also have them designed so that each section of a particular fact sheet is within an accordion. Excellent. One other question, just a little bit on the scope of the project and the site here for reference. What is kind of the overall how how active is the site like annually? And then what is the staffing for updating it for publications, links, that type of stuff. People are trying to compare this to other projects that are out there. Yeah, so we the fact sheets are a primary content type and we have attorney review and plain language review of those annually. We review all the other content on the site annually broken up by quarter. So each quarter will take a third or a quarter of the content on the site and make sure that links are working and still relevant and that forms are still good and all that type of stuff. And then in terms of traffic to the site, I actually got a slide at the very end of that. But we're at about like 40 to 45,000 users a month, I believe, as high as getting closer to 50,000 in some months, which is a pretty significant increase from the traffic that we've had in prior years. Nice. So about half million a year is what you're looking at projecting? Yeah, probably a little bit over half million. But yeah. And then what how much staff time does it take to kind of update maintain this site? Yeah, we have so we have one staff person who is manages all of the fact sheets and that updating. So I mean, that's probably 80% of her time, I would say. And that does not include the attorney review. That's something that our partners do. And then I'd say, you know, it's a little bit complicated because we have a lot more active role now, especially with keeping our partner organizations on referral information up to date. But I'd say it's at least like half to three quarters of another staff person's time to manage the rest of the site. Okay, and all of the publication writing is done externally. It's just a manager to to get people together to do that and review it. Excellent. That's all the questions we've currently got. People definitely have a lot of interest in this topic. All right, excellent. So all very good questions. And one last bit before I move on about the accordions, I think one of the other concepts that we pursue in the use of the accordions to avoid people getting confused or not seeing content is to always offer that initial shot of what is included. So as you can see from this wireframe, the idea is that you get a sense of what the content is in this section with the ability to view more as opposed to hiding all the content by default and not giving the user that quick snapshot of what they might find in that section. So moving on, then, once we get past the wireframes, we get into actually mocking this up. And what we pursued in this case was a new visual language. We're using a new series of family of icons to represent the different types of contents, as you can see from booklets to forms to links. We're looking at ways to increase legibility by giving plenty of spacing between each element. We are looking at chunking content, which is sort of key for how a lot of people digest content on the web, and particularly on mobile. And what that means here is you can see how the fact sheets is clearly visually separated from the other resources as a user would scroll through the different areas of content, they would get a nice clean separation between the elements, which just helps give that eye pause and lets them focus on what they want to find. And again, in regards to visual language, Jenny demonstrated this earlier, but replacing the approach from before, we now have a page dedicated for self-help. And what that is is this self-help library is the resource library. And you can see the different terminology that we are terms that we've settled on for legal categories. And then with any each one, people could find the resources that they could use to help them in their individual case. And a family of icons here as well to just give a quick visual reinforcement. Again, we're just trying to make this as easy as possible for people to find what they need. So I love the icons there and a quick comment was that there was somebody here from transcend earlier, who has a great legal icon project out there that is free, it's available. We just did a major redesign of the LSM tap website and the visual does so much better than the text. I really like this. Good job. Great job, guys. Yeah, yeah, thank you. I will just add anecdotally that that, you know, coming up with icons for for every legal topic can be a challenge. It's a balancing act between making something as easy to digest and find what you need and being sensitive to sort of the nature of the topic. So for example, with crime victims, we wanted something that didn't insult or or, or bring up any negative feelings than were needed. But at the same point, we wanted an icon that immediately explained what was the subject matter. So things like that can be tricky, especially in these types of topics. But I think we managed to do a good job. And with that, we're going to go to rock number two. And I will now turn it over to my partner, Tim, to talk you through the triage. And I have to share the screen with him.  Okay. So for the next couple steps, so Dan, you know, obviously focused quite a bit on sort of the user experience and how we work through, you know, complex content via design to make it useful and easy to find. And the next few sections are going to focus a bit more on some of the big technical rocks or technical hurdles that we faced throughout the project and sort of some of the challenges that we faced, and how we were able to sort of overcome them and sort of how each of these turned out. So Jenny alluded to the triage tool earlier. And again, we don't actually call it that on the site, I believe it's now referred to as the Law Help MN Guide. And it is the central sort of call to action on the site. And sort of as Jenny explained, the impetus behind this was this idea that it was very difficult with so many hundreds or thousands of different resources to guide people directly to, you know, the resource or the service they need. And they were often ending up in the wrong place and getting bounced to another place. And sort of sort of the idea here was, you know, out of the gate, like how can we fix this? How can we guide users to the exact legal help they need, not sort of a vague idea or a guess, it was how can we, you know, present all of these topics in a way that works. And so the idea was that we would present sort of a series of questions presented as a logic tree. And this also, by the way, underwent human language review to make sure that it was sort of flow right and that people could understand the questions they were answering. And as the user works their way through those questions down the logic tree, they end up getting a very specific set of recommended resources first. That's sort of the first step is like, hey, okay, here, you know, based on what you just told us, here's a list of resources in our library that match what you're after. And then from there, the user is able to say, hey, you know, I'd like to see if I'm eligible or qualify for any legal assistance through one of the many dozens of providers in the state of Minnesota that might have services, particular to my issue. So then they go through a second series of questions that Jenny referenced, including things like, are you a veteran? Are you disabled? What is your monthly or yearly income? And then we do a bunch of math and logic on that of our own at the end to try to, again, try to really target the right resources for the right people. And then lastly, with such a complex tool, how can we make it fast and easy? And that was true for both the users of the application itself and the editors in terms of how they might go about managing it. And the reason is that there were a number of big sort of challenges here starting with the fact that we were working with a set of just over 300 legal topic IDs that mapped specific resources to specific legal topics. Those legal topics IDs went across 12 high level legal categories. Those categories match the categories that you saw earlier in the resource library. And within that logic tree, we ended up with over 1000 different possible prompts that the users were adding. So it was a real challenge to come up with a way to both present this to users effectively and to create a tool that made it easy for the law health staff to continue to manage this tool over time. There's also, as you'll see later in one of my slides, there's a concept that we introduced called jump points. And this idea, you know, within these thousands of apps and legal topic IDs, a user may start down a path where they are looking at a housing issue, and they get three or four levels deep, and suddenly it becomes clear based on one of their responses that maybe what they actually have is a disability issue, or a discrimination issue. That allows the application right there to shoot the user over to a completely different part of the logic tree to start answering those questions. Similar with optional jump points. The first example did that automatically. The second one sort of said, hey, you know, based on some of your responses, we think you might actually have a housing issue. Do you want to try from there? And it lets the user sort of jump forward to a new point in the logic tree to start answering questions. It also includes domestic abuse warnings. This was a key thing so that if any response triggered an abuse warning, it pops up a window and directs the user for more information specific to that. Jenny also mentioned this idea of sort of leveraging all of this different data that they had where that used to be a very siloed sort of our goal for this tool was to let's bring in the resources and the services into a single tool that allows people to see all of them based on how they answered these questions. And the last sort of big challenging part of the tool is that at the very end, we did a pilot with, I think, two Hennepin County services, but allowed to take the next step, which was, hey, I'm going to send my information directly to this provider and begin the intake process right here and now. And we're actually now, I think we have a slide later talking about a few additions we're making to that and integrating with some other APIs here in the state of Minnesota. And lastly, again, how do we just make this a good user experience for both the users themselves and the editors? Okay, so the next couple slides are just gonna I'll try to go over these real quickly, but this gives you some idea of the sort of planning that we started with. We really started drawing stuff up by hand and trying to identify where the data was living, how it was going to be organized, how the flow of this application might work. And whoops, over on the right there, sort of, okay, well, now that they've gone through this logic tree and they've gotten to an endpoint, how are we gonna actually present the resources and the services in a meaningful way to the users? Sorry, I feel like, okay, my slides are jumping a little weirdly. Okay, here's the slide I wanted to show. So what you're looking at here is a just one sheet of a I believe there are now 15 different sheets in a Google spreadsheet. This is the tool that allows the law health staff to manage this giant tree of questions across all 12 topics with all 1000 some prompts and you can see here in this jump column, this is where they are able to just in the spreadsheet itself identify where these jump points and optional jump points are they're able to trigger when a domestic abuse warning will pop up based on any given response. And over here in the far left column, you can see these legal topic IDs each time you see that that represents an endpoint, meaning, when the user gets to the end of a specific tree, it's going to return this legal topic ID. And based on that, we are then able to return services and resources that have been have been tagged with that ID. And so it's a very, we think it ended up great in terms of a, you know, a very easy but powerful way to manage a very complex tool. A quick question on the analytics related to the tool. Can you tell how each particular question is performing and what your exit points are from that and whether they exited to a resource or they just gave up? Yeah, so Jenny may have bored to say on this, but that was actually a phase two part of the project where they quickly identified like they wanted to know exactly that like hey, first of all, how many people are actually getting into this tool? How far did they get? Where are they leaving? Et cetera. And so I think Jenny actually may have a slide about some of that later. But do you want to add anything now, Jenny, in terms of analytics for this tool? Yeah, that's been really important for us. And we just finished a six month review of our triage tool. And we use those analytics to feel like identify where people are getting confused or like one of the interesting things is that we saw that the kids, teens and schools topic, people were going there because they had a custody question about their kids, which makes sense if you're not a lawyer. But for us, we want to get them directly into the family law section. So we're able to see you like the most common paths that people are taking, and then use that to make adjustments to the tool to better respond to users. Excellent, very helpful. I want to keep us moving here because we still have a lot to. Okay. Okay. So I think everyone has been happy with the tool and as Jenny's slide will show later, it is getting pretty close to the highest usage in terms of the resources that are at the site. We think it turned out to be a pretty powerful tool that does in fact return relevant results, something I didn't talk on very much as if we also have some very unique sorting algorithms that we use to sort these resources based on a series of questions that people ask, such as, hey, have you already talked to an attorney or hey, are you willing to speak to a mediator and by answering those questions, we're able to push certain results up to the top of the list that in theory will be more valuable to the user. Again, it turned out to be very easy to manage and change in that the law helps staff is able to update that spreadsheet and then we have a tool on the group side that allows us to import those changes and they will then immediately be reflected in this tool. I should also quickly, we ended up choosing to build this using React, which is a front end based user interface library that allowed us to build this in a very fast way for users and that we don't have page refreshes and there's a, if you want to go use the tool, just go to the law help site and click start, but you'll see that we're able to go forwards and backwards and start over and in a very good user experience. Another phase two part of the project was SMS and email integrations so that when a user does get to the end, they have something to take away. They can text or email the list of services or resources that were given to them so that they can then refer back to them later. And lastly, I also mentioned that we started with that very early pilot project and we now begin to expand that to accept two different APIs that allow us to immediately, through a post request, send the user's data directly to, I think it's, we're up to two different APIs where that information then goes immediately into the service providers case files. Just a couple quick screens here to show you sort of how it works once you start. The screen on your left is some of the qualifying questions about how many people are in your household. We actually have a little algorithm that then calculates the FPG number for each user, which is then tied to certain services and resources based on that number. And then over on the right, you can sort of see what happens when a user finishes and they see the services in a nice list. Again, we have some accordions down there on the bottom that allow them to, just from right from the screen, see what it takes to apply, how they contact that person, what their hours are, and when they're happy, again, they can text that information directly to their phones. So unless there are specific questions about triage, I'll move into the next big rock here, which is the provider tools. So again, we don't call it provider tools. This ended up being branded as Loone, which, Jenny, I forget, can you give me the acronym for Loone? It's legal. Yeah, legal organizations, online network. Right, and sort of the idea behind that brand. So let me back up. The provider tools was this idea to leverage all of the services in Minnesota themselves to start updating their own data and letting this legal tool know which services they're actually providing at any given moment, what their limits are, whether they serve people with disabilities or veterans or people who live on an Indian reservation rather than having the law help staff have to manage all of that data. And so by branding it as Loone, I think that was sort of one big first step into presenting this as a tool directly for providers. Let's see here. I'm not sure what's happening, but my slides won't advance. Let me try this. There we go. So again, each one of these big rocks started with a big planning phase, and here it was just like, how can we more effectively connect all of these providers so that they can help each other with referrals and so that they can help us manage all of this data? And that involved an advanced search tool to really allow people with, unlike the end users who really don't have a lot of expertise and maybe this is their first time in the legal arena, these are expert users and therefore we wanted to build a very powerful search that allowed them to very quickly drill down and find other services that they might provide to their users. We also needed to provide a set of tools that allowed each individual service to update their own data, and it included quite a lot of data, including when they have clinics, what the schedule is, what the clinics cover, their FPG financial requirements, where they are located, their hours, all of the contact data, holiday hours, and so on. And lastly, again, how do we get all of these services who to date haven't had to do this to buy into the system and to help the Law Help Minnesota team keep all this data updated and fresh and relevant for users. So some of the challenges for this portion of the project, it turned out to be quite complex and that it wasn't just services, it was really large organizations who often have multiple locations and each of those locations might provide multiple services at any given time and that sometimes they may go away, so we needed to provide tools that allowed people at each one of those levels to update and self-manage this data and to find it. And of course, because it was a more advanced interface, it was how do we build this, so it's really quick if someone's sitting at their desk and needs to help someone with the referral to go use this tool and find the right service. Again, how do we get user buy-in and get people to actually use this tool? And then I just wanted to call out here, I think we'll talk about this a little bit more in the client-vendor relations section, but on our end, I felt like we did a little bit, we didn't plan enough for this, we didn't realize going into the project quite how critical it would be to the project, but we often have tools that allow users to update data and it's fairly simple, but this really turned out to be a very, both important and difficult part of the project. So if we could start over again, I think we would have done quite a bit more upfront planning on this one. But that said, it turned out great. I think Jenny can speak to it a bit when I'm done, perhaps, but I think that people are largely very happy with this tool, as you can see on the right, provides a quick way to dive in through all of those thousand, some legal topics that we identified earlier, search by keyword, service area, and all of the various other metadata that we associate with any given service. Providers have the ability to turn off a service at any time so that if they are not accepting the referrals, it won't show up at the end of that law help triage tool. And again, the data that lives in this tool also is shared with the triage tool. So again, it was another great way to sort of leverage this vast amount of data and make it useful for more than one audience. And then I think Jenny would agree that it's an ongoing goal for them to just continue to sort of push this and to get more and more of their providers active in the system. So here you can see there's just a couple of screens here that show you a bit more about how this information is presented. After a search, right from this one screen, they're able to pull down internal staff notes for any given service, what legal topics they provide, and their contact information, and hours and such that allows them to then quickly send someone else off to that service. Right, and again, design-wise, if you back up a slide, Tim, we're making, again, good use of sort of chunking out the data on the screen. We're using highly legible font sizes. We're showing people what we determined was the most critical information to show them at a snapshot without overwhelming them on each entry for each result. So the critical information in a summary is there, which the client controls the length of the summary, and then a series of buttons or tabs, which you click to reveal additional information as Tim mentioned before, such as hours and locations and such. And by using this mechanism, people can access this additional information while staying on the same screen so they don't lose the search results that they just might have spent time accumulating, as well as some nice visuals to quickly highlight particular terminology that's important, such as the large circle indicating, green circle indicating a clinic. It's all designed to just, again, make it easy and quickly scannable. All right, I'm gonna ask a question here. First, what was the specific fonts and font sizes that you used as the default, and then the, I'm assuming those change responsibly depending on screen size? Just off the top of my head, that's a good question for font sizes. Usually we're shooting for 18 to 20 pixel size or 18 to 20 point size for body text, which is larger than sort of the browser defaults, but what we find is easier to read. Of course, it all looks tiny here on the screenshot, but it is not in real life. As far as the actual fonts we used, we were building off of some existing branding that had been done by the client on another site. So we used a, I think they're both free, or one is free, a Google font for the body and then Montserrat, which I don't recall if it's open source or if it's a paid web font, but those were the primary fonts that we used and font sizes. Next one is in Minnesota, is there a hotline and does it use Loon in some way? You want to let Jenny take that one? Yeah, we do not have a statewide hotline. Different organizations have hotlines that cover specific topics and or service areas, but no, we don't have a statewide hotline, but we do, our state funder has made it a requirement of the state funding that organizations getting that funding to use Loon when they're making referrals so that they're using the information in Loon as they're making referrals outside of the organization and also that they keep their information up to date. And do you find that providers are indeed updating their own information or do you need to prompt them and manage them to some degree? What a great question. I say it's a little bit hit or miss at this point, still fairly new in the broad scope of things. So we have some organizations that are just really on top of it and some organizations have had a harder time incorporating this into their workflows, but my office is actually staffing up soon to provide more staff support and prompting and guidance to help make sure that everybody is keeping their information up to date. I think what you're seeing there is very common from what we've seen from other states, especially when you deal with some of the smaller VLPs that may have a lot of turnover or very little staff, that they really need that community manager to help them. A question back on fact sheets really quickly. How many attorneys help review those fact sheets that you send out? This is like two or three or is this 100 plus? What does that group of editors look like? I don't have an exact number. Our processes that our fact sheet manager will, for example, send the housing fact sheets to our largest legal aid organization to the housing unit and have attorneys just sign up for the fact sheets that they're gonna review. And if all of those don't get taken, then we'll send it to the next legal aid organization. So I don't have an exact number, but it's definitely more than two or three. Okay, so it's definitely a process that kind of manages the community externally. Exactly, yeah. Awesome, thank you. I'll just add really, really quickly there too for those of you who are not from Minnesota, just the whole loon acronym that is our state bird. So there's a distinct Minnesota reason behind that name. Correct. Okay, I'm gonna spin through this one really quickly because I know we're, I think we're running over where we had time to be at this point, but sort of the last big technical route for us was this idea of a multilingual site based on the fact that a significant percentage of law helps users are either low income immigrants or people who simply don't speak English. And so the idea was, what can we do with the site content to make it available to those people in the widest way possible? And that started out with simply defining the languages that we used. We ended up with four, defining which content to translate and who will translate it and when. Some of the key challenges was making this as easy as possible for editors to manage, and perhaps more importantly to allow them to manage this over time. We certainly couldn't hold up lunch to wait for every single piece of content on the site to be translated. So the idea was as translations come in, they get added to the site and then magically appear for any given content. And then there were just a few technical hurdles. We ended up translating basically the whole interface. So all the menu items that you see and buttons and dropdown filters all had to be sort of translated in custom string files. And then we ran into the Hmong language in Minnesota is very popular. I think we still have the highest population of Hmong immigrants in the country, but strangely the Drupal translation system didn't have a culturally appropriate language prefect. So we ended up having to do some custom work for that. I just wanna add to him really quick too that if you can just say a few words just about how Drupal as a content management system makes multilingual work so well. Yep, I was actually gonna do that on this slide. So this, you can see the interface for languages where here we have our English screen and Drupal basically has a long history of providing good. It's always been an international project and therefore they've always really worked hard to providing translation tools. And in particular Drupal eight and of course nine and beyond have really come a long way in terms of how powerful the system is to add these translations to any given node. And therefore for users of course to then interact with that content. So at the end of that again, we ended up with a full interface translation meaning it's not just the content, it's the buttons and the help tips and the dropdown filters and all of the other things that you might see on the site. Full content translations and I have an asterisk there because as of today, the triage tool we looked at earlier has not been translated and I think that was largely a fact of what a burden that would be for the law health staff to have to translate thousands of questions four times over. So that may be something for the future but other than that the vast majority of the site is available in English, Spanish, Hmong and Somali. And we did end up with again that flexible option where as translation comes in the law health staff is able to add them to the site without holding up other parts of the site. And then down there at the bottom again this presentation will be recorded and the slides are there but our developer who worked on this wrote a three part series on how to really do this and do it well in Drupal 8 if you're interested in some of the technical challenges.  Okay let me catch up to where we were in the slides and as far as time goes we're actually right on schedule so I think we're doing well. What we're gonna touch on here now is the subject of the other rocks that we had alluded to earlier in the presentation and then really just run through the different phases of the project and some insights and lessons that we can share with you. So just to go back to where an earlier slide we begin the project with multiple rocks and this shows our list that we began with and like a lot of projects when you go in you identify those critical ones but sometimes as you get into the more realistic portion of how this is all gonna work time wise and budget wise some have to be moved to a future phase or left behind and so the ones in blue are the different items that we did address as part of the initial phase. There were a few items there in black for intake and SMS integration which are both being addressed in our phase two work with law help. So talking through project phases we alluded to some of these elements as sort of interwoven in some of the slides we've already discussed but to take you through our process in a brief manner each project we began with a phase of strategy and analysis and what that meant for the law help project the law help project was defining those project goals and outcomes including our rocks, how we can measure success and we'll talk about outcomes for the site in a later slide and then how can we sort of plan and understand what we need to make this an effective redesign and some of those tasks in both content strategy work and this particular slide I'm going to just touch on the work we did for site maps that's the site navigation, the competitive analysis looking at other sites and a creative brief. All right so when it comes to site map what we're doing are one of our overall goals is always to simplify and to make this easy for users and increase and improve the user experience. So we're looking at, we audited the previous site we considered the terminology that was being used we looked for ways where the number of links could be reduced, where we could rename a section if possible to make it clear for the end user and how that affects both our site navigation for mobile users and desktop users and also the specific terms that maybe our resource library for example or might be categorized as or the different results that you found in the loon tool how can we use Drupal's taxonomy to tag these terms appropriately so that users can find the results they need. So when we were looking at the original site map you can't really see all the details in the smallest screen but we took time to understand what the site was doing where it was sending people how it was organizing content and in the screenshots here you can see in the upper portion the the menu structure here is somewhat similar in that we keep a very short navigation there's really two primary choices to be had just like in the previous site that we renamed these and repurposed them but then the site also had other ways to access information in the sidebar that appeared on every page including things like donate or legal forms and then the footer had quite an array of navigational items including multiple ways to access those things that we talked about earlier of the resource library like fact sheets or forms and letters and so what we did is we looked at all this and thought how can we simplify this so that there isn't a confusing redundancy or complexity to the site and what we ended up with as you've seen in earlier screenshots is a new navigation that is really straightforward there's two primary calls to action menu buttons the guide and the self-help library we make it very clear that there's the donate and about us pair that down the language adds additional complexity to any site so at least visually we separated those choices on the top and then for the footer as well we put those secondary items down below as before but we greatly simplified what was shown here so as to make this easier again for the end user when it came to competitive analysis what we like to or as I call it comparative analysis sometimes in particular in this case because these aren't necessarily competitors to what law help MN is doing but they are colleagues and similar organizations so we like to look at what they're doing right or at least what we think is working well and what is not and what lessons we can learn from that so we look at sites like the Illinois site or the Connecticut site or the Michigan site and I'm looking at this in terms of how are they organizing their site content how are they presenting calls to action and how does this work for what we're trying to do and we looked at this from a visual standpoint we looked at the sites from a technical standpoint when it came to things like the triage tool and determining how we're going to organize and build those for the new law help MN site and lastly at the end of the strategy and analysis phase we present a creative brief this is our roadmap for moving forward once we get past the planning phase into design and site build identifying those key problems to solve our minimum viable product the MVP what is the minimum thing that we have to deliver above all else and this is a document we can reference as we move forward into future phases and revise as needed just real quickly as you can see with the MVP and a minimum the site must offer a triage system provider portal multilingual and resource library and aka these were the big rocks that we covered so we did nail that one and get our minimum as well as expand beyond that once we've got our strategy and analysis behind us which by the way included all the technical planning that went into those tools that Tim referred to earlier we get into the visual redesign and consider how we need to approach that so notes from the RFP was that the design needed to be clear, friendly, bright and official and the official keyword there heard of references at least for me and I think what it was is how can we make this site feel trustworthy these are people that need legal help they want to make sure that they feel confident that this is a resource they can use and the way you can reinforce trust is of course in the content you're offering and your reputation offline it's also a clean and consistent user interface will certainly add to that level of trust as well as considering how we can make this site optimized for mobile users people all over the state using accessing this information on different size devices and also how can we address accessibility to make this reach the broadest possible audience so as you had seen earlier we wire frame we think about fields that we require for each one how do we prioritize if we're laying out a page for Lune we need to first orient the user on the page that's the banner that's first the links to the side of that message for my account those are secondary but we want to make sure they're accessible in a logical location so we style these these are unstyled mockups but let's us have conversations with the visual on how we're organizing our site content and just to let you know we're down to about 15 minutes and we've got about five or six questions in queue so I will speed up a bit for wire frames to design direction so just real quickly one of the things that we do at Electric Citizen we don't like to go straight to a full-page mockup assuming we know all the different components and how they're going to exactly lay out across a site especially a site of complexity so what we do instead is take a component-based approach we come up with a design language that can then be applied once approved to different components of the site which are then compiled into pages so what we see here is only a portion of sort of a broader design board that we present to the client and have conversations and once we agree that we have the design direction and the right feel for the site we take that direction and start applying it to mockups so you can see how the wire frame on the left became a page mockup on the right there are some similarities between the two but there are design choices that get made along the way and to meet accessibility guidelines we have to, what we recommend is using a combination of automated tools and some of the more popular and recommended tools that we've used include the Wave Accessibility tool and the Axe Accessibility plugin these are both free tools that you can find online as well as manual testing and for us the manual testing means browsing the site using a keyboard, testing the site through speech text-to-speech tools that are offered for free on the Mac for example and running through a series of rules that we apply to make sure that the text is legible that content is chunked properly So a quick question on this so with user testing we really go out there to try to get clients to test when it came to accessibility testing did you have testing done by users with disabilities or an organization that employs users with disabilities to do that type of testing? Right, so that's a great question and the answer is no we did not that is definitely if you are able to budget the time and resources for is the optimal way to test for accessibility in lieu of that we rely on our knowledge of accessibility best practices and these online tools that help us to audit the site So as you can see just a quick screenshot of how the Wave Accessibility tool runs through the site and gives you results are on the right the Axe tool inspects the site for any warnings or violations and into user testing then segue to that question how we conduct user testing we defined a series of tests we tested the Lune tool, the triage tool the overall website and the mobile experience this is something that we coordinated with the client to find subjects for Lune it was a matter of finding legal service providers who are willing to volunteer their time to come into the office where we conducted the usability workshops for things like testing triage and the overall website Jenny and her law help team advertised to find subjects that were willing to participate I believe that they offered a small incentive to people that were volunteers Yeah, we worked with some of our partners so legal aid offices had flyers in the lobby that public libraries had flyers and then at legal clinics also did some advertising and offered a $20 tariff card for each person who would come in and do 20 minutes of user testing Yeah, and so the results of our user testing where I would document the results meeting with each subject and summarize for the client you were able to identify adjustments to the user experience that were needed were able to tweak the tools and then occasionally identify bugs that we hadn't uncovered yet that needed to be fixed by our developers some sample comments, for example in one of the tests people were supposed to find something in the city of Burnsville, Minnesota and the tool head that we had built was organized to search by county they didn't know what county Burnsville was in probably because they didn't live there so that leads to small tweaks like we offer a link under the county finder to that links to another webpage where people can easily look up a county for a particular city we make adjustments to once purple had completed the legal triage, it wasn't cleared to some where they should proceed so we adjusted how we presented the final results between finding a lawyer and previewing the self-help resources but overall a lot of the user feedback was positive and it was really beneficial for any project if you're able to reserve time to interact with people prior to launch to make those necessary adjustments Okay, we're down to about nine minutes here I'm gonna cruise through a few quick questions and then we'll get back if there's other stuff still to cover first one, what was the overall budget on this project and how long did it take from your beginning of scoping to launch user testing? Well, I can say that for the length of the project it was a year-long process from the time we first met to the time we launched I don't know as far as Jenny how much we wanna share as far as budget goes but I know that that was the length of the project was at least a year Was this funded through a TIG grant? Yeah, we had multiple funding streams for this We had a TIG grant for about $150,000 and then we also had money through the Justice for All Implementation Grant and money from our state judicial branches technology fund Multiple funding sources definitely kind of the common strategy here Did that include your time or was it that plus your time? The funding primarily went towards vendor costs but we did have some funding for our time Excellent So there's a comment here which I'm not sure the best way to put it which is any tips or any ideas on how to better to help the inter-organizational collaboration that it takes to put together all the resources, that type of stuff and they also mentioned that it might be worth having a separate training kind of talking about that in general because it does, it hits this project but it also is much broader and I would definitely suggest that it's a possible webinar topic for early next year because it's pretty big but tips on how do you get all these different stakeholders working together on a project this big for the triage tool stuff and for Loom I think it was really helpful for us back before when we were still just applying for funding that we had the JFA strategic planning was a really good forum to get everybody from like social services workers to standard civil legal services organizations and get all those people in one room and on the same page about what our overall goals were in Minnesota we also have a meeting every other month with the civil legal services partners within Minnesota and then we had a steering committee for partners who were directly involved in managing this project so people who were named on those three grants that we mentioned got together every other week to discuss any like bigger decisions that we needed to make on the project. Tim, you mentioned a guide earlier not sure exactly which one it's referring to but which guide were you talking about earlier and do you have a link to that? I've moved you up to be an organizer just in case you got a link or a reference that you can share with the audience there. Well, I guess I'm not quite clear on the question but I think maybe the question is referring to the Law Help MN guide which is if you just go to lawhelpmn.org and click start here on the homepage that I believe is the guide I was referring to. Okay, I'll let the person clarify it because it's been a few minutes since we got that one coming in. Okay, in the search top right there is no box to type in search term. Does the search icon take you to a search page where you type in search terms or does it expand out and just become a search box? Somebody's asking about kind of the functionality of that and looking at the site which I can put it on the homepage.  Okay, excellent. I think that cover, oh no, she was asking about translation guide. Oh, okay, yeah, that was a, I don't have the link handy but it was on the slide, I believe the third slide of the multilingual portion and let's see here. It's on our website, electricsitizen.com so if you go to electricsitizen.com and go into our blog, I believe it's our most recent blog post and it's a three part series that sort of walks you through all of the challenges and ways to approach multilingual in Rupal. Okay, excellent. Yeah, there it is. Thank you. That covers all the questions that we've got so far. You've got about four more minutes here. If you've got any additional content or takeaways that people should really consider in kind of writing a grant, a proposal, that type of thing to do something like this. Yeah, and I'd like to just jump in and make sure that we have a chance to go through Jenny's last couple slides which may be helpful in that discussion in terms of just the results that they've seen. I agree. So we will go right into project results. The site launched at the end of January this year and here's a series of slides that show the, some of the results that we got from launch until today. I'll turn that over to Jenny. All right, let me scroll through here. So we've had a huge increase in usage since we launched. And this chart shows just the increase. A lot of that is due to organic search activity where I think that our SEO is just much higher with the new site so my help is showing up higher in results so we've been really thrilled with that particular result. We also, as I mentioned earlier, have analytics that show both what options people are clicking on when they navigate through that triage through the guide and then where people are dropping off. So we've had, and these numbers are a little bit old, but we do have a significant amount of traffic going through the guide and people are generally, the majority of people are able to get through to the self-help resources. So that first, we call it like phase one of the guide. We do see a little bit of a drop off and people going from that page where they get their fact sheets and other self-help information into the screens where they enter eligibility information to be able to get referral information. So that's something that we're continuing to tweak with Electric Citizen. And we are able, we have a 10 hour a month budget with Electric Citizen to continue to make improvements to the website which has been hugely useful in trying out new things and continuing to improve it. And then my last result slide is just to show that the guide has consistently been within the top five most visited pages. So we're really happy with that. We still like to drive more traffic. We see that most people who enter the site don't enter at the home page, they enter at a content page. So one of the things we're thinking about is how if somebody winds up the rights and responsibilities of unmarried parents on that booklet, how can we make sure that they know that there's another tool that they can use to find other related resources that might be able to help them? So overall, we've gotten really good feedback from users and a lot of, I would say in terms of making sure that your project, you're able to communicate your success and your progress to funders. It's been really useful with Electric Citizen that with each phase of the project, they've kind of produced a deliverable design direction or wireframes or different actual documents that we can then use internally to review but also share with funders to show them the progress that we've been making with the project. So two last questions here as we're into about our last 30 seconds. One is the analytic tool that you're using. What analytic tool is that? So I'm gonna jump in. Yep, go ahead, Jenny. Oh, I was just gonna say Google Analytics that Electric Citizen has customized a great deal for us. Awesome. Yep, it was a challenge to get that, to get some of the analytics into the law help guide but that's what we're using. So you created a custom kind of dashboard for that? More custom Google Dimensions and metrics. I do believe we have a dashboard for them but it was more just like, how do we reliably determine where someone's leaving and coming up with good ways to track some of that stuff. Excellent. And as this is a highly customized Drupal site, if somebody else wanted to kind of replicate this project and do something similar, how much of this is transferable to another site or how much would you guys be going back to the drawing board? Boy, that's a tough question to answer. It depends on how closely another site would want to adhere to exactly how the Law Help Minnesota is doing it, I think is probably the most accurate answer. It's because so much of the system works together in terms of the resources and the services and the loon tool all working with the Law Help guide, it would be very difficult, I think, to take out pieces of that. So I don't know, that's actually a great question but very tough to answer. We did work with or at least talk with several other groups across the country, I think Colorado, I forget which one, Michigan, and looked at ways that we could potentially leverage some of their work and it turned out that it just simply wasn't close enough to the specific goals that Law Help Minnesota had to really take advantage of those tools. Okay, well, thank you all so much for putting on this presentation. I really liked the effort that's going into the community to improve the statewide websites and this was great. Any resources that you guys have as panelists, please feel free to email those to me that you would like to include it with the blog post. We should have a recording of this up within the next few days. And then we've got over 20 more, or not 20, over 10 more trainings throughout now from now to the end of the year. So please take a look at the upcoming ones, especially the two that we've got on security that are out there and share it with people in your program and come to those future events. Thank you all so much. Thank you. Yeah, thank you.