My name is Jillian Thiel, I'm from ProbonoNet and I wanted to welcome you to Language Access Strategies for Legal Aid websites. Before I get into introducing today's panelists and the topics of today's webinar, I would like to go ahead and turn it over to Sartreau if there are any additional introductory comments. Three quick things. I put down in the chat a link to our training calendar for the training events for the rest of the year. We've got four or five webinars coming up. I also put a link in there to our YouTube channel and tap videos where we have over a hundred past videos. We archive all of our trainings and they're up there and online for free. Additionally, the slides that you're going to see today are under the handout section. You can click on the slides and download them and follow along for today's presentation or to have them as a reference later. Thank you so much for putting this on. Jillian, look forward to this. Great. Thank you, Sartreau. So today's presenters, we've got a great group with us today. We have Christy Cruz from the Northwest Justice Project, Joanne Lee from the Legal Aid Foundation of Los Angeles, Maria Menlin from Transcend, Dennis Rios from Illinois Legal Aid Online, Sartreau himself will also be discussing a few points around translation evaluations, and last but not least, Angela Tripp from the Michigan Legal Help Program. And during today's discussion, we are going to cover a bunch of topics around language access for statewide websites. We're going to be talking about, we're going to have Angela Tripp starting us off with discussing nuts and bolts and how you start, how you execute a translation project. And then we will also have Dennis Rios talking a little bit about various considerations and other lessons learned from translation projects. And then we're going to go, we're going to turn it over to Christy Cruz and Joanne Lee to talk a little bit more about best practices and best practices as well as considerations for sustainability. And then we're going to talk, we're going to talk with Maria Menlin around the latest translation updates and machine translation. And then last but not least, Sartre is going to talk a little bit about evaluating translation projects. And as mentioned earlier, we really hope to encourage a lot of discussion and questions during today's webinar and presentation. So feel free to enter in questions and comments to the chat box. And we look forward to this presentation and to having you all joining us. So I'm going to go ahead now and turn it over to Angela Tripp to get us kicked off with nuts and bolts of translations. Excellent. I'm going to make sure I show the right screen. Hello, everyone. Thanks for joining us here. My name is Angela Tripp. I'm the director of the Michigan Legal Health Program. And I'm going to talk about how we built the Michigan Legal Health Mirror Site in Spanish and just sort of, like Julian said, the nuts and bolts of how we do things. So the other speakers are going to cover a lot of other aspects. So I'm going to really focus on how we decided what to do and then how we executed it. We built Ayuda Lagal to Michigan in 2012, about a year after we launched Michigan Legal Health. We had a very specific mandate from our funder to build a Spanish language version. So we went with the Mirror Site. And that means that everything on the two sites are identical, except that not everything on Michigan Legal Health has been translated into Spanish yet. But everything that has been translated is exactly in the same place, operates the same way. By the end of last year, we had translated six videos, 54 toolkits, and all of the contents of those toolkits, which is hundreds of articles and frequently asked questions and checklists. And three of our do-it-yourself document assembly tools. We continue to add new translated content as quickly as we can. At this point, nearly all of our content, except for the do-it-yourself document assembly tools are translated, and I'll talk more about that later. By the end of 2016, there were just shy of 70,000 page views of Ayuda Lagal, and that makes up only 2.1% of all the page views on Michigan Legal Health. We did not translate the organization's or court's pages into Spanish unless they very specifically and capably offer services in Spanish. To do this, we hired an outside contractor to build out the Drupal language module, but we maintained that in-house. We hired a new bilingual staff attorney who manages all the content on Ayuda Lagal. We issued an RFP for the translation work back in 2012 and hired transcend translations who have translated nearly all of our content. We're often asked if we're going to build mirror sites in other languages, and at this point, we have no plans to do that. We don't have the resources, and I hope after my presentation, you'll understand just how many resources you need. So next, I'm going to talk about the nuts and bolts of translating content, and this is every piece of content on Michigan Legal Health, whether it is an article, a frequently asked question, a checklist, or an organization page. Everything has the option to build a translated, paired piece of content. This is a screenshot of what the back end of the website looks like. You can see we have this English article about being evicted, and it has the Spanish article is linked there with it. Building this ties each piece of English content to its translated partner, which is important because the website is built on a concept of tool kits, which are collections of articles and frequently asked questions and forms that provide a user with everything they need on a specific topic. Because everything is paired, we don't have to rebuild or reorganize content. It's automatically where it's supposed to be. It's also automatically tagged with all the proper legal issues, tags, triage tags, editors, search for motion, everything, all the metadata that's in there is already, we don't have to redo all of that. This also helps us keep track of what has been translated and what hasn't, and what's been published. Our administrative assistant keeps track of every piece of content on the website and whether it has been translated. This changes all the time as we publish new content all the time. She sends content out in batches to transcend for translation. If it's older than one year, our staff do a plan language review before it gets sent out. If it has been published within the last year, it's already recently gone through several rounds of plan language review, so it just goes right out. Transcend translates it using their records of our preferences, word choices, and history. They send it back. Our bilingual staff attorney reviews everything, makes a few modifications, answers questions that Transcend leaves us as notes and gets it ready for publishing, and then the administrative assistant publishes. We use all human translation. We don't have any machine translation built into our systems at this point. The nuts and bolts of maintenance. This is a huge part of responsible language access. Like all content on a site like Michigan Legal Help, the Spanish content must be constantly monitored for updates and constantly modified. Our site's built in Drupal, and so we built a system in Drupal to simplify this task for our staff. Whenever a piece of content is modified, our bilingual staff attorney gets an email notification that he needs to update the Spanish content. That's automatic. Anytime anyone changes anything, even if it's just a period or a misspelling, that triggers a notice to our staff attorney that that same change needs to be made in the Spanish version. What you see on the screen here is on the left, and the yellow is the prior, the old version, and on the right is the newest version. You can see the colors highlight the paragraphs with changes, and the specific word changes are highlighted in red. When our staff attorney logs in, he can easily see what's been changed and then modify the Spanish content so that it matches the English. All changes to content are initiated in the English versions and mirrored in the Spanish versions. Maintenance is one of the most time-consuming aspects of supporting language access. Be sure to plan for it. I believe that having a bilingual person on staff is a requirement, and it is also a requirement to build solid systems to track your updates and translations because otherwise there's no hope of keeping everything straight, especially when you're using bilingual content. The nuts and bolts of translating forms. When you start translating forms, your first big decision is whether to do a monolingual form, which would be one form in English, one in Spanish, in our case, or a bilingual form, where one document has both languages on it. There are pros and cons that each approach, but after much debate in Michigan and working with the courts, we decided to go with monolingual forms. The court made certain forms available on their website, and then we asked them to translate additional ones so that we could translate our document assembly tools. And this was one of the biggest pain points in our process, was translation of forms because the court wanted to control the translation of the forms because it was putting their stamp of approval on it. However, we disagreed with a lot of how the court's hired translator did the translation. They didn't really pay attention to what language we used on Michigan Legal Help. They weren't terribly consistent within their own documents, and they weren't really open to our suggested edits, which is what you see on the screen here. So this was a big struggle for us and something that was a surprise to me. And managing the politics and the financial aspects of this disagreement and how to resolve it was really hard. And the court had no Spanish speakers on their staff to weigh in on our concerns. And so it was sort of what we said versus what their translator said. And so what we have are forms that don't really match the language of the website. And there's not really anything that we can do about that. And believe me, we did try this. We spent months and months and months on this. As a practical matter, people get one set of forms in English and one set of forms in Spanish and are instructed to file and serve the English forms. And we do a lot of watermarking on the Spanish forms to help them understand not to serve them and not to file them. But the problem is that any judge-created order does not get translated. And the best that we can hope for is that people can match up a blank Spanish form and a filled in English form and figure out what the order says. It's a very bad system, but the courts don't have the capacity right now to translate all judicial orders. And the judges aren't comfortable signing orders that they can't read. So we're hoping to continue to work on improving this. Translating interviews, on Michigan Legal Help, we use both hot docs and A to J author to translate A to J author interviews. We print out the script. We highlight the pieces that are text and answers and have that translated and then rebuild the interview in Spanish. For our hot docs translations, we use the handy-dandy text management tool that we built through an LSC TIG that's available to everyone with access to the ProbonoNet library. It extracts all the interview text, puts it in a Word document like this so though you can add your Spanish language here and then run the tool again and it will place all the Spanish language or the translated information back into the interview where the original text came from. You still need to do a lot of testing of the interviews on both cases, but it makes the work a lot quicker. I'm gonna speed through my lessons learned. A lot of people are gonna talk about the importance of plain language, so I'm gonna skip that. Definitely plan for exponentially increased maintenance and be selective about what you pick to translate. Focus on high-use ones. One's without a lot of free text responses because people have to enter those free texts in English unless you have a translation system set up and that can be challenging. Look at your data and be flexible. Remember that only a small fraction of your users are going to be using these interviews so if a small fraction of a small number of users is a really small number, so again, focus on once they get a lot of use. Also, choose interviews that don't change frequently. We've been working on a divorce interview for I think two years now and it just is a bottomless pit because it changes every month. Definitely coordinate early and often with courts, particularly where they have control over your forms. A few more. I think you really need to bring bilingual staff on board and then brainstorm other ways to increase capacity for big projects such as testing. That was one of the other problems with the divorce interview. It was too big for one person to test alone. Because of limited capacity, brainstorm other solutions for language access. One of our solutions is to try to open bilingual self-help centers because we can't do another full version of Michigan legal help in Arabic, for example. Again, this will be covered later because it's a different regional dialects. Be consistent in your translations. Whenever possible, have all the potential tools you need at your fingertips. One of the problems that we had with the court forms is that we were willing to create court forms ourselves but we lacked the special tools that they used to do the layout and the type setting of their forms and that's really important to them and they weren't willing to do that for us and so we weren't able to create our own court forms. Finally, systems and checklists make everything easier, more reliable and less prone to error, especially when you're working with multilingual content. If you are embarking on a translation project and wanna use any of our checklists and systems as examples, please reach out to me. I'm very happy to share what we've come up with. And that's it for me. I'm gonna change the presenter to Dennis, right? Yes, I'll go ahead and do that. I'm gonna pause briefly. Oh, okay. Thanks, Eddilla. I'm gonna pause briefly to see if there are any comments or questions here. I am seeing a couple resources posted by Claudia Johnson that I'll share out with the groups. There's a blog on a text management tool and also some background on the Law Help Interactive HD text management tool. I'll be sharing those around shortly. And then I see a question also from Caroline Robinson that asks, is Translation Management Module the Drupal Module Michigan uses and what kind of file do you send to translate? Do you send a HTML file? I believe that's the Drupal Module we use, but that's not my area of knowledge. So I might be wrong. I can check on that and get back to you. We send all of our stuff to Transcend in Word. We experimented with HTML in Word and for a variety of reasons, it worked easier for us to send them Word files and for them to send us Word files back. Great. And I'm seeing a couple other questions and requests for a copy of the steps and checklist. So I'll make sure to put you into contact with those folks. Perfect. And... Actually, if you can get a copy of those, I can also post those to the write-up that we do on the webinar. Yes, I will do that for sure. Thank you. Great. I'm seeing a few other questions, but I'm gonna go ahead and move along to Dennis and make sure that we have some time at the end as well for additional questions. Thank you, Angela. Absolutely. Thanks. Good afternoon, everyone. My name is Dennis Rios. I am a content manager at Illinois Legal League Online and I am in charge of managing our Spanish website and our Spanish resources. Just as a little background, I've been translating for around 20 years. I have a law degree from Mexico and an LLM from the States. Now, I wanna share a couple of considerations in our experience in dealing with this translation of our content into Spanish. And much of it is very similar to what Angela just shared with what they did and had a legal help from Michigan. It's because we learn everything from you, Dennis. Not true, not true. So, as with any product, I wanna emphasize that it's very important to know who your users are, their preferences, how they communicate, what their needs are, and especially, you know, this is especially important when providing legal assistance and information. Now, specifically to Illinois, as you can see on the map and as most of you, I'm sure know the most spoken language after English, Spanish, and this is true in 43 states in the US. Here in Illinois, the Latino population represents about 20% of the population and most of them come from Mexico. There's also folks from Guatemala, Colombia, and Salvador. Now, Spanish varies depending on what part, you know, of the world you come from and what part of the US you're living on. So, if you're developing content in Florida or New York, for example, it would be very different from the Spanish content that we're developing here in Illinois or, for example, California, where most of the population might be from Mexico.  For example, my family from, my mom's side is from Mexico, beans, you say frijoles. In Venezuela, they say caragotas. In Puerto Rico, gandules. So, it's really interesting how much, you know, the words differ for something as, again, as common as beans. My partner who's Venezuelan, you know, we obviously both speak Spanish, but, you know, it's fascinating how we don't even call bananas or papayas the same or even the mop or the broom. So, I've literally had to put up a list of Venezuelan words on the fridge so I can improve our communication. And the other thing to consider as well you know, I've spent a lot of time in Mexico, I went to high school and to law school there and I've lived in a couple different regions in Mexico and even within the city, sorry, within the country, people refer to, you know, different things differently. So, even, you know, the word for car, you know, in parts of Mexico they say carro and in other parts they say coche, mueble. So, even those things vary and again, if you can sort of determine, you know, those preferences and those characteristics to the population that you're serving, you know, your content is gonna be that much better. Now, let me tell you a little bit about how our English website became bilingual twice. The first time was back in 2012, which was when I came on board to develop this project. And so, the first initial Spanish website was developed with a TIG grant, the Technology Initiative Grant from LSE, that was done through LAF, Legal Assistance Foundation of Metropolitan Chicago. And that website included videos, automated forms, text content and it also included a Spanish live chat feature for the first couple of months. Now, some of the lessons that we learned from this first Spanish product was that generally Latinos prefer video content. They use their phones tremendously, so most of our, when we were taking a look at our analytics using Google Analytics, we realized that very quickly, Latinos were using their mobile phones much more than just English speakers. And so, that's why we determined that it was essential that our website would be mobile friendly, which again, this version was not the old version. The other thing that we did learn from this was that the Latino community here in Illinois was not a big fan of the chat service. Our chat service on the English website has been very successful. And again, we did launch it in Spanish in that first website, but we didn't really get much of a response from the community. I'm not sure if it's, there's a lack of trust in regards to chat services or just, there's a different preference on how the Latino community communicates, but despite several efforts to inform and to promote the service, we couldn't really get it to take off. So we discontinued that service. Last year, we launched any platform and July will say that we copied Michigan because we did a triple platform, which is now mobile friendly, which is the same platform that Michigan has. And again, we did this because using the data that we had, obviously both just generally the population is now more and more going towards mobile content. But specifically again, the stats show that every time the Latino community the numbers are greater for the Latino community as far as mobile usage. The current website, the new platform, was launched in February of this year. We've locked around 4,000 page views for the Spanish resources and we have around 600 articles available, which again include general legal guides, videos, forms. One thing to consider when you're managing multilingual content is sort of the way that you manage your resources. Now this shows a little bit what the sort of process that we use to translate is. So generally, you have a draft or review, you do a plain language review and a style review. So again, so the content is consistent. You then publish, you submit the content for translation. You promote the content on social media, emails and other venues. And then you evaluate the sort of impact that those resources have. Now it's important that before you start to translate a content piece that the English version, the English version is accurate enough to date that it's been formatted, indexed, any style editing changes have been done and that it's gone through a plain language review. This is because you want to make sure that that version that's being translated is final so that you don't have to go back when a change is made or if it's updated or if it goes through another review, then that's just gonna cost you more money, more resources, more staff time. So if you can avoid all of that, that is going to be one less headache that you have to deal with. Now here at Illinois Legal Aid Online, we use a mix of machine translation and human translation. Now our translation team is composed by myself, content manager, a Spanish translation coordinator Vista who manages a pool of Spanish translation volunteers. Now the way that works is that we recruit Spanish translation volunteers from the US and around the world. The Spanish translation Vista does an evaluation of the volunteers and then after she assesses them and their skills, she does the training and then kind of works with them as they provide translations and does a review of their work. Now I have to, I do want to emphasize the importance of having the human review done by a qualified translator.  You want it to be accurate. You want the language to be consistent. But also, and one of the things that I think isn't, perhaps isn't discussed that much is that, for example, for our English content, we always try to have a six to eighth grade reading level. I think the same thing should be true for the Spanish content and it has to be, not only accurate, but it has to be readable. And for example, we've actually been looking for resources that can help us assess the readability of our Spanish content. And I'm not sure, we haven't really found a whole lot of information on that. So if anybody knows of resources that they might be willing to share as far as how to gauge or improve the readability of the Spanish content, I would appreciate if you would share that with us. So I'm not gonna go into a huge whole lot of detail on machine translation and the services since a couple of the presenters that are presenting today will go in depth into the subject. But just generally I wanted to mention sort of the machine translations and software and services that are out there. So one of those are the computers as the translation tools, which is basically the software that helps translators with their work while they're translating. Now there's different companies, I think probably the biggest ones that are out there are SDL, Trados, Smartling. We initially were using Lingotech when we first launched this new platform, with Drupal. Right now there were some limitations with that platform so we're taking a look at some other options that might best sort of suit our needs. The other sort of services that are available and that again other presenters are gonna talk about it more in-depth are is machine translation and memory translation. Now machine translation I think you've probably heard of. Most of us are familiar with that. And memory translation is, we had initially when we started working with Lingotech, we were really interested in the memory translation aspect of the software which what we understood initially we thought it was that the software was actually learning from the translator as he was correcting a translation for example. So it was registering the preference in words, the preference perhaps in sense and structure and it was getting smarter as it again learned from an actual human translator. This actually turns out to be not the case. Memory translation basically what it is is that if a person, a translator translates a sentence and then that exact same sentence repeats itself in another translation then it gives you the option to replace or to just plug in the translation of that exact sentence that you translated in another content piece or another article. So you can gain a benefit from this if your content sort of repeats itself. If it's repetitive and phrases or lines are going to repeat throughout your website then that would be a big benefit. For us we are trying especially with this new website to avoid duplication of our content. So we're trying to avoid duplicative content. Just like Angela mentioned, it is important to plan not only for the resources that you need to develop your Spanish resources or your garden as we show on this slide, but also how to maintain it. And I think that a lot of times that isn't taken into account and having a good multilingual website takes a big effort, staff time, resources, significant funding, not only to launch but again to maintain and much like a garden generally it's easier to develop, to obtain the resources to develop the content but it's generally harder to find the resources to maintain that same multilingual content. So that's something to keep in mind as you're developing these resources. We currently again only have the content available in Spanish. The idea is to expand in the future into Polish and then Chinese, which are the other languages that are most common here in Illinois after Spanish. But we're still working on obtaining the resources to develop those other languages. And yeah, basically that's been our experience. Again, there's my contact information if anybody has any questions or comments. And thanks again for attending. Great, Dennis, we do have a few questions from the audience. One is if, and I think this is, we could open this up to anyone, but just a question about the cost of translation services specifically and then any other comments that you have about the cost generally of undertaking a translation project. Wow, I mean the translation, I mean the cost is very tremendously. I don't have sort of a number of generally, the general cost that we've incurred with translating the website, but I mean a human translator, a professional translator probably charges between 10 to 25 cents per word, I would say. Another thing, and this is going back to sort of the resources that I was mentioning earlier as far as software. So we've been doing research on different softwares and again machine assisted sort of tools to help us again, to help us increase our capacity and also talking about how do we get to the point where we can develop resources in other languages, what we need to improve, we need to expand our capacity and in order to do that we're looking for other services and technology that might help us get there. But for example, there's services that we've looked into for translation management systems that basically you, it's not just sort of, the offer memory translation, the machine translation within their platform which is a CAD tool or computer assisted tools, but you can, you basically upload on the cloud all your documents that you need to translate it in many different formats basically, most of the formats you could think of and then from there that allows you to then assign the different content pieces to volunteers, but in this case, and again the company that we were talking to was SDL, Trados and Smartling, from there they offer a pool of translators that you can access from this platform or this cloud. And so the benefit of that is that, if you don't have a, for example, if you don't have staff, a translation sort of department or staff in your office, they can, they provide the whole platform and management of translators and just directly upload the translation once it's done. However, the quotes we got from them started at around 50,000 per year, between 50 and 100,000 a year just to use the platform. That doesn't include the cost of human translators and other features. Now again for us, since we do have a sort of system where we have in-house translators, it doesn't make a whole lot of sense to use those services, but I just wanted to share with you sort of what's kind of out there in the cost that you might incur if you're interested in some of those tools. This is Angela. So we, like what Dennis quoted for the price per word is we're between 22 and 25 cents per word. We do have a staff attorney, but he only probably spends 25 to 30% of his time working like dealing with the translation in Ayuda Lagal and the live help and the sort of the ancillary services. And then our administrative assistant probably spends 7% of her time, maybe 10% of her time sort of facilitating the translation. So those are the costs that we've had. Great. So we have a number of other additional questions coming in. I'm gonna ask one more and then direct and then if we have time at the end, we'll get to these other questions or I'll put you in contact with the specific questions. But a question here on if you do video, a question from Caroline, if you do videos in oral or audio content, what dialect do you end up using? And once you get funding for Chinese, how do you decide on traditional or simplified Chinese specifically for written content? Huh, we, as far as the Chinese content, we haven't really gone that far into those considerations, but I'll make sure to jot that down that that's something to consider. Yeah, we haven't, again, we're still kind of working on the new Spanish platform and website that we launched. So that'll be a little bit further down the road. But as far as the video content, and again, the question was what sort of language was that? What, how do you determine what dialects you'll use? So again, we tried to determine, you know, most of our population here in Illinois, Spanish speakers are from, from Mexico, I think for the most part, the areas of Mexico that, there's folks from all, you know, all over Mexico, Guadalajara, Chihuahua, Chinchocan, so Central, Southern Mexico, Oaxaca. So we try to keep the terms, you know, as, as, you know, as common as, as general as we can.  And also for, you know, the, the other Latino populations, Guatemalan, Colombian, you know, we want, we want to keep the terms. We're not, we take, you know, we consider obviously that 80% of the Latino populations here are from Mexico, but we wanna make sure that, that other 20 is also, you know, represented and that they can understand the resources that are on the website. So it's, you know, it's sort of just putting on a balance, sort of the different demographics that you have while, you know, developing the content and the language. But again, I do wanna emphasize on the importance of having the Spanish and plain language. Now, what I say that sometimes when you get a translation, and even, you know, for myself working in translating, if I get a translation, you know, back when I, when I used to translate for firms or the court, if you get a translation of a legal document, it is much easier to translate that legal document and to direct sort of the appropriate accurate Spanish legal wording than to, do a translation that you have to provide for the general population. You know, so when you get a document done by an attorney, you know, that's not plain language that it, you know, the easiest thing would be to translate it into, you know, normal accurate Spanish legal language. But the very hard thing is to translate that content into Spanish plain language. So when you're outsourcing the content to a translation company, again, make sure, like Angela mentioned, you know, they've been working with this one company, they know what their preferences are. The words that they use, the sort of wording and style, that's important because not only is it important to use a qualified human translator, but you wanna make sure that they know the style that you need for the website. Not just that it's accurate, but that it's plain language and that, again, your population is able to sort of digest it easily. This is Maria and I can respond to the Chinese question, if you like. Yeah? Sure. Yes, please. Yes. So I think you really need to know your community. And so for instance, here in California, if we were going for Southern California, it would definitely be a simplified Chinese characters that we would want to use. If it were in Northern California, it would be probably both because it's pretty much a split between mainland China and Hong Kong, Singapore, Taiwan. And also, I think we need to keep track of what's going on in the world. Increasingly, Hong Kong is changing, Singapore is changing as mainland China changes. So that language is in flux and what's happening with Chinese is in flux. So we need to understand who our Chinese communities are, what they're reading and what they're speaking. And we only know that by keeping in touch with that. Great. Thank you all for answering the questions. So I'm gonna go ahead and transition over to Kristy Cruz and Joanne to talk a little bit more about best practices and sustainability and keep your questions coming. Okay, thanks so much. This is Kristy and Joanne and I are both gonna be talking about different components of best practices in translating content and that really applies to both online resources, but also you can extrapolate that to the work we're doing in our legal services organizations with our LEP clients. And first I wanna introduce you though to just a group that there were both members of the National Language Access Advocates Network. This is a national organization that supports and engages in effective advocacy to eradicate language discrimination and to promote language rights. You can, if you're a legal services attorney, you can join that group on the Pro Bono Net website. If you're a government agency or an employer contractor of a potential opposing party, you wouldn't be allowed to be in that group. We talk about different strategies, but there are a lot of public resources. And so I wanna mention that because these are your colleagues, many of them are working in your legal services organizations and they're advocating for effective communication access to services like courts, like emergency management and all the wildfire work that we've been really engaged in that lately. We are working with hospitals to ensure that they're complying with their Title VI obligations with labor and industries to make sure that injured workers are accessing services. And so then we sometimes have to look at our own services. And so your inland advocates interact with these diverse communities and see the impact of language barriers and we wanna be a resource to you all as well. And so let me see if I can move the screen forward. There we go. Okay, so the first best practice I wanna talk about is setting a standard for translation quality. That's in terms of both quality of the content and translator qualifications. In terms of the quality of the content, I think many of my co-presenters have already discussed these considerations, but I used to be a sign language interpreter for many years before I became a legal aid attorney and an expression in that work is garbage in, garbage out, right? So if I am trying to interpret something, but the meaning is not clear, I mean, obviously an interpreter can interject, but it's not the job of the interpreter, the translator to really clear that up other than this engaging and interactive process where there are questions, but you don't just sort of pass it off and expect it to get sort of cleaned up in the translation. So I will leave the content piece, I think we've covered that and I know we're running short on time and focus mostly on translator qualifications. So I do need to back up and talk briefly about translations which are taking written word from one language to another and it really is a unique skill set. So someone who is bilingual in two languages is not necessarily trained to be a translator or an interpreter, but here we're looking at translation of a written document from one language to another. This skill set does require excellent knowledge of two languages and above average writing ability, reasonable familiarity with the subject matter and also a willingness to familiarize yourself with terminology and use of a glossary and I think we'll talk about that in a little bit what we mean by that. But it really, and it is this interactive process, right? A skilled translator will go back and ask questions and seek clarification just like a skilled interpreter and sometimes with interpretation, if it's going very smoothly, you may think, wow, that was a really great interpretation but it's really the quality interpreters are ones that interject, ask for clarifications, look at a dictionary and so we can sort of apply that also to translation. In terms of qualifications and certifications, they are very different. So interpreters, often in our work, we're looking at interpreters who are certified by our state court systems or the National Center for State Courts Consortium testing but with translation, this sort of the one that's often referred to is the American Translator Association. They certify in about 15 language pairs and when that's not available, you can look to other certifications, including interpreter certification by your local courts because then they at least have the familiarity with the legal terminology. There are also resources like the Interagency Language Round Table, round table, excuse me, it's ILR, that's a federal interagency workgroup that brings together Department of Defense and lots of other entities to look, they're engaged internationally and use resources in many languages and they have some assessment tools that you can look at if you're looking at assessing bilingual staff members' qualifications in terms of their writing level, their reading level, their speaking level, all of the different components of doing this work and I encourage you to look at that. The next best practice I'm gonna talk about is developing a translation protocol and this involves a process, I made this little chart and there's probably lots of steps in between each one of those but it was getting a little out of hand so I made a shortcut here. Just in terms of developing that original content, of course it's going to be in plain language, excuse me, that wasn't supposed to happen. And then we're gonna move along and make sure that it gets reviewed and it's going to go through an editor and a review process. I think later on I'm gonna talk more in depth about this community review and another opportunity to modify the document and I think we just had a question about this which is really when you're looking at what language is or having some assessment of quality, it really does help to work with your partner organizations and maybe even have some focus groups around that. I do wanna talk briefly about an example we have here in Washington State, NGP, we worked with our state court system to establish a protocol that includes a system for what process the courts will go through when they're doing court translations and it requires the use of an ATA certified translator in a language if it's available or an interpreter who has five years experience translating and then in that case you would also prefer them to be a court certified interpreter certified by our Washington courts because that tells us that they have some familiarity and assessment on their legal terminology and they understand the ethics involved. The protocol also discusses the use of a primary translator with an editor and a reviewer and it also requires the translators working through doing court translations to use centralized glossary. Next I just wanna talk briefly about some existing standards and resources that are available to really help inform this work. The first one is LSE program letter 0402 and this was issued back in 2004 and these are standards that were developed that require LSE funded programs to set competency standards for interpreters and translators doing work in LSE funded programs. This would also include assessment of the skill level and competency of your bilingual staff members if they are involved in this process. The next resource is the American Bar Association Standards for Language Access in Courts and this document it's very comprehensive, it's not only looking at translation but that is one of the standards that it does cover and it establishes guidelines for courts providing translations including qualifications of translators it warns against the use of machine translation and some of these same concepts can apply to legal services so encourage you to look at that resource. Every time I need to see it I just Google the ABA standards and it comes up and then finally the third resource that you might wanna take a look at is this document created by the Consortium for Language Access in Courts which is called Translations, Getting It Right, A Guide to Buying Translations and then the Guide to Translation of Legal Materials which you see there. These are all documents that cover topics around competency, best practices to ensure in the end that we get quality translations. I really feel like both Michigan and Illinois, Antelope and Dennis have highlighted this next standard which is maintaining control and really staying involved in the process of translations, right? Just like we maintain control of the client interview through an interpreter we wanna maintain control of the translation work and not just delegate the entire process out to a vendor. Partly because having centralized coordination of our translation work can assist in reducing costs and it helps maintain our standards. If we're setting a policy then we wanna make sure we're involved in that process. We can keep lists of translators by languages for sort of internal work. We can keep lists of dark translated documents and then we wanna maintain and update glossaries of the language and the terms as they've been translated so that we can have consistency across documents. We also want to gather our demographic data annually and I think that's been asked a few times like how would you know? And part of that is really doing an assessment every year and sometimes it's by county because really the demographic data changes by location around the state and so we wanna be responsive to that and sometimes our field offices can provide us with that. Courts can provide us with some information. We often in Washington we do a cross analysis by courts, DSHS or you know you're sort of TANF programs, school data around home language use and then we overlay that with the census data to try to figure out if there are languages that we need to be more responsive to and engage with those communities. Okay, I gotta move a little more quickly. I know we are over time. I wanna talk briefly about the process of using software tools, right? So these are both and Dennis talked about this a little bit. This computer assisted translations where you would have your translation memory software and then machine translation. And what I would say here is you know, the best practice if you are going to use machine translation would be to do that with human review. Machine translation alone without human review should be avoided because really we don't have a way of knowing that information is accurate. And it can be something as simple, I think Dennis's example was great where it's a simple word beans, but it can be translated in a way and it's not just that it could be wrong but it could cause confusion, awkward language can undermine our work and our ability to serve clients. And so this standard of using machine translation with human review is supported by the standards that I've mentioned. And when we start to try to sort through content and group it by legal and non-legal content, that can get really confusing, especially for a staff person to try to decide that. Is this legal content where you should have human review, non-legal content? And so the best practice is to avoid that, try to decide that and use, only use machine translation with human review. Just briefly we were gonna talk about that ties into this idea of programmatic integrity and making sure that the services we provide are appropriate services and we're not providing lesser services to individuals because of a language barrier. And I think Joanne was going to just provide some comments on the next two slides before we transition to her slides. Hi everyone, this is Joanne Lee from the Legal Aid Foundation of Los Angeles and I just wanted to echo what Chrissy has been talking about in terms of this debate that's been going on within the legal services community about the use of machine translation and whether human review is needed and she went over a lot of really important points. But I think this issue of sort of programmatic integrity and making sure that we're not having a discriminatory effect on LAP communities is really important. We've talked a lot about plain language and what we do just to get a good version in English, we would never just sort of pop it into a machine and take whatever it comes out and put that on our website. And I think that's the same. We don't wanna have a substandard level of what is acceptable for English speakers versus people who don't speak English. And I think some communities would find it very offensive if we have this sort of substandard where we're okay with some grammatical errors at best. And then of course at worst, we are putting out information that is incorrect or could lead to mistrust or a lot of other issues within communities that are already very difficult to reach. So we wanna make sure that any material that we put out there is very clear and welcoming and is nuanced in the way that we could really do proper outreach and provide services to them. We do have a couple of questions coming in around resources that I'd like to get to before moving on. So one is from Megan. If you can't afford a mirror site, is there an appropriate minimum translated amount of information or process and or is anything less considered a failure of language access? And then dovetailing with that, Carrie asked obviously manual human translation and translation management is ideal, the eventual goal, but if you have much smaller resources is providing Google translate the best option until potential funding and RFPs are granted for manual human translation. So I'm gonna cover a little bit of what you can do with limited resources and then next bit of advice. Would you like me to do that? Yeah, that would be great. Okay, yeah. So I think I need the screen. Sure, I'll go ahead and make you presenter, Joanne. And this is Christy. I'll just briefly say that last slide, Joanne is also gonna mention that in her piece, so it'll work perfectly. Around community engagement, yeah. So I hope you can see my intro slide and just to move it along, I know we've covered a lot of information, so I'm just gonna give some basic tips on planning and sustaining, especially as the person who asked the question, David, you don't have a lot of resources to build a full mirror site as Angela and Dennis have already talked about. So I think one really important thing to do is make sure that you have a plan. And I think it's really overwhelming when you think about adding multiple languages to your site. Obviously, if you have one that's very prominent, such as Spanish in a lot of states, you do think about building mirror sites. But if not, you wanna sort of have a plan and a timeline for expansion, obviously, create a budget. But if you have these limited resources, you can start with basic static information, things where you can fill in addresses, location for clinics, you can summarize and prioritize certain information that you can give importance for those communities. And then you can also have links to PDF, so it's easy to update and format. Links to in-language videos, PSA. So I'm gonna just show you one example really quick. If you can see this, this is a local domestic violence shelter that we work with quite often. And if you go to the homepage, you'll see that there are these languages listed and they serve various communities, so they felt they needed to have it in many languages. But as you can see, if you click on one, it's not a full mirror site and they just have a few layers here. They have sort of an intro to their services, they have a video here that you could click to, that's a PSA in Korean, how to get help, background on domestic violence and sexual assault. And this is how they've chosen to try to provide basic information, but then also have it available in multiple languages. So I'll go back to this. So that's an example you can look at. Also, if you don't have the technology to build in sort of like translation memory and all of that, it's good to keep a glossary even if you're just creating it yourself on a Word document of organizational name. Your own organization could come out in different formats depending on what sort of machine translation you put it through, or even individuals may translate it differently. So you wanna have a consistent organizational name for yourself, for your partners, for what you refer to for the courts, other government agencies, and then key terms and phrases, as well as legal terminology. I'm just gonna give you an example. And I have the honor of being on a translation subcommittee for the California Courts Language Access Plan implementation task force. And online, there are a lot of resources that the Judicial Council staff have worked very hard to develop. I think some of them are on the call, so shout out to all of you. And you can see this is a glossary. It's in 10 languages. They have their recommended fonts and other sort of symbols that you might be able to use. So this is available online here that you can take a look. And also if you go to this website and go to Language Access, you can see a whole toolkit of different materials, the translation protocols that the courts has developed, so it's very useful. Another thing that you might want to keep is a database of translation history. And so similarly, you could have some sort of Excel document where you have the name of the document, the language, the data it was created, the author, where you have it, especially if you have multiple links in different places. The data of the original translation and the translator, and then all of the updates, as well as who the reviewers were, and any state-sensitive information after which you may want to pull that information down, and then also have a plan and timeline for future translations. And then finally, as all of the presenters have mentioned, you want to make sure that you budget costs for updates because sometimes updating is even more work and more time consuming and more expensive than putting it up originally. You want to make sure if you're not doing it through some sort of platform that any final versions you put on the website are formatted properly because the fonts are very challenging to work with, and you may have slight spacing that creates a completely different word.  You want to reach out to the community by doing focus groups or even informally checking in with community groups you work with, other legal organizations, LEP community members, and then, as others have mentioned, having trained bilingual staff is really important, making sure they're qualified and trained, but also just having them review the material that you have up on your websites and also other materials that you're developing. And just a final comment on whether the use of Google Translate is better than nothing. I use Google Translate quite a bit just to double-check items that I'm translating, words that I'm translating, and I have to say that a lot of times I get things, as a result, that are completely not at all what I'm trying to say or the opposite of what I'm trying to say, and so I don't think that it's better than nothing. I think it's dangerous to rely on Google Translate, that's my opinion. So I don't recommend that you use that in place of translations. I think if you have limited resources, you can start with sort of summarizing your basic services, putting it on a PDF file and linking that on your website until you have the resources to expand. This is Angela. I second that. I don't think it's a good idea to just put a Google Translate link on your website, and I think what you could also do is prioritize a small area of content to translate if your budget is only that small. I agree with just smaller projects. This is Dennis Rios. Yes, definitely don't do that. I wanna say that there's this whole idea where some information is better than nothing. Well, wrong information is way worse than nothing. So you don't wanna do that, especially with legal information. So yeah, I mean, if you wouldn't do that for your English users, definitely don't do it for this community that's particularly vulnerable and really needs content that is accurate and readable. This is Maria from Transcend, and I would agree with what everyone has said, and I would think if the users know that they could use Google or Babel if they want on their own, and I wouldn't do anything to facilitate that because of the reasons that people have already articulated. Great, so I think we're at a good transition point actually to Maria to talk a little bit about getting into this question of translation and using systems, so let's go ahead and transition over to Maria. Thank you, so I get to talk about machine translation and I just wanna mention how I came to learn a little bit about machine translation. About 15 years ago, I had the honor of meeting the chief of Google Translate in Menlo Park, and so I wanted to be prepared and understand a little bit about it and read up and not be a complete fool when I went to meet with him, and my so-called brilliant idea was that plain language was gonna be one of the new Google Translate languages and which didn't work out for reasons we can talk about later, but it was really very enlightening and I was able to learn about the basics of what goes into machine translation and how machine translation really varies a lot from one language to another. So let me tell you what little I know about machine translation. Okay, and all right, so this is a topic that of course as a language professional, it comes up a lot. We have people calling us every day and say, oh, can't you just machine translate it and then sort of fix it up a little bit and charge us half as much as you usually do and so we have to go and explain why it is that we can't do that. So we decided that it would be really great to team up with some linguists from the university and to really make a very professional research study in a variety of court languages to figure out how usable in fact would Google translated text be for a court scenario. And so we rated it for overall understandability, native-like syntax, which is a good predictor of whether someone is actually gonna start reading it and continue reading it for the accuracy of the key messages and for reader confidence. So by the time you finished reading it, would it be like one of those sort of goofy translations that you get on the back of something you think, oh my God, I have no idea what this means and just throw it out or yeah, okay, this is probably a good translation. And then just what we talked about right now is machine translation better than nothing. Okay, so let me tell you, we're just in the starting stage but let me tell you what we know now. And we did this for Spanish, Armenian, Korean, Vietnamese and simplified Chinese we're starting to look at. Armenian is really easy to talk about because Google translates into quote unquote Armenian well, there is really no Armenian, there's Eastern Armenian and there's Western Armenian. So there you go, you can't really do it. And if you don't know that there are two distinct languages, you would be really confused by putting the text in and getting this language that's really not one or the other. So we weren't able to go forward with that. So the first thing is really what is your language pair? Google translate does and we're just talking about Google translate, we're not talking about, I'm sorry, I'm getting a little bit jumpy for some reason in the slides. Does a much better job on some languages than it does on others. This is also true of other machine translators but in the US, most people use Google translate. So the best source language we believe and this is a guess, is North American English. And the best target language we think is one romance language, probably the least inflected European romance language. The worst languages are probably all the other languages we have in the United States that are high volume. They're not really suited to Google translate. They might, you might have better luck with some of the other machine translation tools available. And the difference is tremendous. You might do pretty well, for instance, with a romance language and you might get something just so dreadful or laughable, for instance, if you looked at Korean or Cambodian. So it's a huge range. The next variable would be what kind of content are you trying to translate? Are you going for small lexical units? Do you just wanna find out about little pieces that you're trying to put in an application like a name, address, phone number? Or are you looking for a longer text, like a legal notice? If it's just small pieces, you might have a better chance and that could also depend on how inflected your language is, how complex it is. Some languages like Armenian, for example, have a high degree of inflection and case and in every instance in which they appear, they have to be properly inflected and so this is something that Google Translate won't do for you. For other languages that are less inflected, they will. But for instance, if you were doing Chinese, like the question we had from Caroline before, do you know which Chinese you want? Do you know what the difference is? Do you know what your community wants? And are you equipped to be able to evaluate if this is the right one? Google Translate might offer you four or five. Do you know which is the right one? So there's a whole series of questions that you can answer yourself on this first Google, ask yourself on this first Google pass. For the longer texts, this is something we took from one of the courts. This is actually a, this is not something that we invented. This is a real language access notice that tells you what the policy is. And the reason, this is pretty long and something that I just want to share with you. We've had all of the wonderful speakers talk about plain language. I think that probably the court considers this to be pretty plain and I'd like to see it a bit plainer. And so the plainer it is, the better Google Translate would be able to do with it in any language. And another part would be that any time a word would have more than one possible meaning, you have more opportunities for mistranslation. So in this particular slide, I highlighted the word appear because our Vietnamese translator told me he had a really great laugh because he said it translated the word appear like I dream of Jeannie just sort of snapping her fingers and all of a sudden she appeared in court. And so that was a nice way of getting to court. And for all of the languages also a simple word like ticket, it ends up being like a movie or a concert ticket rather than a traffic ticket. Something like hearing, it can be your ability to hear rather than a court hearing. So words that we look at and they think, oh, well, these are pretty straightforward. They're really not straightforward. They're quite fraught even though we talk about the improvements with the neural capabilities of Google Translate, it's still not that easy. One of the things that we can do is to make things easier and shorter and that does improve, I'm sorry, this is jumping around so much, the ability to get a better draft. But I'm guessing, I'm guessing I know that no one speaks all of the languages that we have in the United States and you don't know which languages are gonna trigger these problems besides the fact that Google Translate is written with errors always in some of the languages and some of the phrases. And so there's gonna be problems everywhere but to minimize the problems, you would have to make things shorter and simpler. And but even very short and simple words would have problems. And I just tried to think of examples in the languages that I use that just very quickly, ticket, clean, clear, tight, file, license show. I mean, just everyday words like that would have problems in Google Translate and the list just goes on and on about the type of things that would be very problematic for Google Translate. Okay, so quality assurance, something that everyone's talked about. As a language agency, we wouldn't use Google Translate. I've heard other people say that, they would use Google Translate. I don't think that if you make this decision, it would have to be reviewed by a certified translator if you're using this text for some legal purpose. It would be very dangerous I think to use Google Translate and to assume that the text was okay. And so I think it's really important to understand that there are really big risks to think about, you know, the other choices and options that people have talked about and to understand that there's, you know, why we're seeing that there's really significant improvements and constant improvements in what's going on with Google Translate that we're just really far from it being reliable in the legal context. So the factors that influence the final anti-product are the language pair, the size of the content that you're translating, the simplicity, plain, very plain is better, the review process that you have, and of course, understanding the limitations and knowing how, what steps you have to take to remedy them so that you can produce a certified, reliable translation. And that's it for me. Great, thanks, Maria. And then I'm going to turn it over to start to say a few closing words on evaluating translations and rounding out and closing us up on the webinar. Excellent, thank you so much.  There is a lot of fear here with regards to what is going on with computers and that this technology is going to be misused and abused. One of the things that I would like to point out though is that I really like what Miranda is doing and trying to objectively evaluate how this technology works and give us some targets for computers to move towards. Alan Turing really came up with the foundational idea for deciding whether or not a computer is in that area of artificial intelligence and that is really, could the computer be sent to think if a human tester cannot tell it apart, if the results that the translation software puts forward actually passes the standard test that individuals are putting forward for certifying someone to do translations. And that's the type of science that needs to be done to objectively look at these advances. In other areas, things that we thought were impossible, chess and go, we have already seen that this technology has moved forward very, very quickly. Language is very difficult, but there's an opportunity here to look at and examine it and set benchmarks for technology and then continue to evaluate the technology based not only on what it is doing by itself but how it works better in conjunction with others. One of the biggest messages from this whole discussion today is something that Jerry Kasparov talked about a while ago and also came up in the idea, this sci-fi book, The Peace War, which is the idea of using computers and humans together for better results. So far, we have seen that when we take human grandmasters and pair them with chess experts, chess expert computers, the resulting play is much, much stronger. And in the medical field right now, we're having objective studies that show that not only are computers able to do diagnostic at equal and sometimes even better levels than human doctors, but when you take human doctors and add in their expertise to the computer results, what we see is much higher accurate test results. So I believe that as we start to objectively measure computers as they continue to go on and improve, we will find not only that the results will improve and will improve drastically in the coming five to 10 years, but that the combination of humans and computers are going to give us the best result, which has really been one of the key points throughout this presentation today. And I'd just like to open things back up to any final questions. I know that this was one of our longer webinars. We've had a lot of interest in this particular topic and we've had over 120, 130 people here today. This is Kristi. I'll chime in and say one quick thing following up on Brian's head. There are some existing test, objective tests that support what we're saying now, which is machine translation does not meet the standard of a human translator. That's not fear-based, that's Google's own research that shows putting a simple sentence in does not reliably get you the results. That doesn't mean we won't be there in the future, but, and I look forward to making sure that we can do those sort of assessments, right? And go into that thoughtfully, but I encourage people to find those. I find them online where there really are studies that are looking at sort of accuracy levels at the current state of things. This is Maria again. I think that really specific examples can be helpful. One of the things, one piece of feedback we got from our Chinese translator for that evaluated simplified Chinese text was that Google Translate did a terrible job on saying you have the right to an attorney. So that's pretty straightforward and short. It sounds like a piece of text that Google Translate would be able to handle well, and she said that it did it terribly. So I think it's a good example of something that, we do have to be careful in our hopefulness. There are some things that Google Translate can do well, and there are some things that it just cannot do well. And I don't think we should ever use it without review by a certified translator. And especially because we have tools like translation memory and translation management, which can be so much more effective and efficient and save money. I think that there's other types of technology that can leverage what we've got that empty machine translation is, it could be pretty dangerous. Right, and yeah, I agree also. Again, I think that a combination of machine translation with human review is badly well and helpful. And to Sartre's point, I don't think it's fear. We have to be clear about what things are and call them by their name. I will be ecstatic once the technology is at that point that you can just rely on it. I mean, that would be amazing. It would make my job so much easier. I'm the first one to jump on board, but we are not there yet. So, let's be clear about that. I think we definitely- We're writing things in English. We are also not there. We have review processes. We go through plain language. So, I think there will always be review processes, but how much and what quality and what can be added is where the bar is changing continually. Right, right, but like I said, I think the consensus with anybody you talk to is that, and again, we are using machine translation and I do see the value in it. And I think that, of course, I mean, machines think so much faster. They can do things in a heartbeat and they're improving our capacity day by day. And so, again, I do see the value in that, but again, I just wanna make sure, because I keep seeing this false notion that, and that question coming up, that you can just use machine translation. So, again, I think, again, the value, it's great, but right now, with a combination of both the human and the machine.