Thank you. Hello everyone. So yeah, I will be talking about evaluation. This is the work that we did and the framework that we created for evaluation of different machine translation engines. Couple words where we're coming from. So, my colleagues are here also here in the room. We work in a company called Optum. So it's a big healthcare organization and we are working in a team that basically built work with innovation inside the company. So we try to risk different machine learning technology and make them usable for the broader organization. And part of like one of the work that we did was around machine translation. So machine translation is basically task of converting text from automatic to converting text from one language to another. It has a lot of application and big enterprise. So it can be like machine translation when people just translate, I don't know, some text just to communicate with each other or it may be some large scale batch translation when we translate a lot of like marketing materials to other languages or our product to other languages and so on. Or it can be like part of chatbot. Yeah. So our goal was to understand what technology is available out there, what we can use, what's the quality wise. It looked like and because we work in healthcare domain, machine translation are usually like common models are usually fine tuned for like general domain of text. Our texts are more specific. So another question was what will work better like on our type of data? And just before your use of machine translations, why basically be doing it? Because machine translation is a hugely evolving field. There was like, it comes through several stages but for recent years with all the development in you know, like deep learning models, multilingual models, they get a huge boost. So the new models coming up like each year, each month, even so it was important to us to have a framework in place that will allow us to test new models as soon as they can come in and compare it with our previous models. And just to give you perspective, the huge as the field is huge. So if you want machine translation, there's a lot of like different models available. So there's a lot of things to come to choose from. And that's why it's important to like be able to evaluate them between each other, choose the best one. So as I said, basically our goal was to develop a framework that allow us to choose our own data sets to choose different engines and different type of machine translation engines and compare them between each other on different metrics and basically see how well they're performed. And to be able to have this framework that we can basically run frequently, agent new models, agent maybe new data sets and so on. A little bit about implementation details, so what we did. So first one is like my personal like problem is language. So if you work with machine translation, we need to know like what language we're working on. Like from what language to what language you want to translate an example here, it's three models from Facebook made by the same team like each one like a year apart. And to set the language, say like for each model they use the new like type of the language. So it's in X6 and in Latin and so on. And if you like start going deeper, it's actually more complicated because like there's a lot of English languages that we treat the same, but then Portuguese in Portugal and Portuguese in Brazil as probably different languages, Chinese languages are definitely like there's two different languages actually. So the problem is like how we can understand them if you work with different engines. And I saw like a lot of examples when people just do like get first two letters in lower case, so please don't do this, it doesn't work. Like luckily there's people over there who not only complaining like me about the problem but do something about it. So there's a library called Lancotes. That's all it's for you. And the way we use it again, we try not to think more about what language people pass to us. We just like we have a model that have some a list of supported languages and whatever format they prefer. We have a language that like you want to evaluate. So we just try to function that finds the closest language and then just use it. Next piece is the actual translation engine. So when people think about translations, they usually think about in terms of like we have a string, we translate the strings that's it. But if you need to translate a large amount of data and especially if you want to do it for evaluation purposes, there's a lot of more stuff to do. So you need to parse the file. You need to skip text that was already translated before because again it may be a long process that may fail at some point. So you want to restart not from the beginning but from the point where you don't have translation. You need to pronounce the text. You need to do sentence separation properly if your text is not sentence separated. Then you need to split it to batches. Then your engine come in and translate the text and then you need to put it back together, maintain the order of the lines, like catch errors if there are any. And again, like huge batch translation may take a lot of time. So we want to catch errors, just output them and not stop the whole process because it's really annoying when you in the morning check your batch script and it just falls. So to support all this, what we did, we actually created the base translator class that take care of all of this stuff like for us. And then for actual translation engines what we need to support is just three functions. So initialization when we basically load our model or like authenticate to cloud engine or whatever, then actual function to translate the lines. And at that stage, we make sure that the number of lines that come into the function is actually like suitable for the model. So if he says that like how much batch size must be no bigger than like four lines, then we make sure like we make sure outside that no more than four lines come in at the stage. And another function to set language pairs. So it's usually important for multilingual models that may translate between different language pairs. So again, just like a model specific way to set the language. And by doing so, basically using this architecture allow us to use different engines. So we're not limited to like let's say only transformer models or only cloud models or like our own APIs. But we can easily create adapters to any model that we have so we can create adapter for Azure that just sends a request to API. We can create adapter for our like local transformer model like Mary and we can use our local APIs as well if you want to like compare the engines with what we have like in-house and so on. Next problem was data sets. So again, when we want to test machine translation on what data set we want to test it. The story here is usually when the new models come in is that there's several like common data sets from usually from workshop on machine translation from different years, but they're very limited. So they're like English to German or English to Romanian by Romanian for some reason. So it's maybe good to like to present scientific results like here's the data set our model is better like how great are but not good when we want to evaluate the models between each other like on from different angles from different domains and so on. Likely there's resource for this already. So people in Opus they create public storage of all like not all but most common translation data sets and they have very good APIs. So basically you're just saying like I want this data set and translation from English to Spanish then they give the data set to you and the data sets are very big. So what we do on our side, we just support it like downloading this data set extracting like because they usually like big like million of lines extracting some test data from them randomly and so on. So in the end basically what we end up with we had our data sets when we have our source data our reference target translation and then we just had translation for different engines and compares with each other. Another problem that we try to solve is about metrics. So there's basically there's many more there are multiple ways how you can properly translate one sentence to another language. So when you have the reference sentence it's a question like how we compare like what we had from a translation engine to what we have as a reference. So any metrics that you have will be approximation. For most of the metrics again there's third party libraries but they usually have some question like how to use them and so on. What we did in the end because again created a pluggable framework when you can use different metrics we used several third party libraries to use like six most common metrics. In the end turns out that in our use cases we can just use blues so the blues are like most common metric for machine translation but because we usually care about like huge difference in quality. If you like different metrics are important if you want to improve the model a little bit. Yeah, so for metric again we just create a base class that you need to fill in. Just say like do you need to take inization? Do you need like source or just target in reference? And yeah, what your languages are. Okay, so and then for evaluation basically what we did we created just three scripts, three scripts. So to download datasets translate them using different engines and then evaluate the results. And everything is managed by a hydra configs. So what you do like if you want to just reproduce our result you can just call the scripts without any parameters. If you want to do it on your own on any other dataset from Opus you can just create the configs that will say like this is the name of the dataset from Opus. Like I want 10 lines out of it without duplicates and with minimal length of string like of such characters. And the script and then when you run your scripts you just set like datasets to download with my new quick evaluation set. And then you pass it on to the next scripts and everything will be done for you. Again, if you want to evaluate on your own datasets and that's the stuff that we do like in-house because in addition to public datasets you have our own, you just create your dataset on your own whatever means you have and you just provide again the hydra configs that say this is my dataset, like name private dataset this is the source file, this is the target file like number of lines and that's it. And then you obviously doesn't need the first step now to download and you just for the script you just say it like translate datasets this is my private dataset and then evaluate it. Again with the engines like if you want to support new engine you create your like translation class that will be inherited by base translator class implement all the three functions like initialization, translation and setting language and then put it in config saying like this is my class like called custom and it should be like you need to put this name like in the enumeration inside the code and this is some settings for my engine and let's go. And again, like just in the script you just say that my translation engine is my private engine. Okay, so the next piece is basically evaluation that we did. So we choose several engines to evaluate again for our, like from our perspective that was interesting for us. So first of all we took two cloud engines Azure and Google, then we used that we evaluated Merian MT, so Merian is a very good architecture for machine translation. It's originally was written in C++ so it was very fast, it is very fast. Then it was converted to like Hagenface Transformers we actually use Hagenface implementation but because architecture is very good it's one of the advantages of Merian is the model are very small. I don't know, I don't remember. It's 300 megabytes, something like this is the model. Another model is NIMA, so NVIDIA have their own natural language processing suite with different models for different use cases like speech recognition, summarization and so on and one of them like, so you have machine translation as well. And then we also tested three models from Facebook. So MTM 100, MTM 50 and no language left behind is the lightest models from Facebook that was actually released last week. And many thanks to great contributors to Hagenface who just supported it in a week. So I had a chance like tomorrow or yesterday to plug it in again to our engine test. And yeah, we include multilingual models here basically for multilingual models, usual use cases like why Facebook is working on them as they care about low resource languages so the languages that have low amount of like parallel data that nobody trained specifically like translation engine. In our use case we do evaluation for translation from English to Spanish, it's like most common translation pair. So we didn't expect much from this multilingual engines but we feel that it's important to test them anyway because when you go to other languages in your pipeline it's much better to use one model instead of like the difference of if they prove, if they may prove like useful, it will be good to know. Another piece was data sets. So again, as I said for like public evaluation we just use public data sets from Opus. One of them was actually in healthcare domain. So in me it's a corpus of documents from European medicine industry translated to different languages and other was like some extract from Wikipedia, some TED Talks, Open Subtitles, books in Bar-Crow and CC-aligned, it's aligned text from Web-Crow. And for each data set we try to extract around 5,000 lines for each it's like depends on the data set because some of them have a lot of like for example especially like Open Subtitles they have a lot of duplications, they have a lot of short lines so it turns out smaller but other data sets we use around 5,000 lines pretty stable let's say for evaluation. But the data sets itself are actually quite bigger like much, much bigger like a million of lines. You can use them like if you want bigger evaluation. Yeah, and the metrics so there are different of them. We mainly focused on blue and I will show the graphs why? Because it turns out at least in our use cases the metrics was where aligned between each other. Two interesting ones, so the blue is traditional metric. Yeah, you have here. So basically it's a number between zero and one that measures similarity of machine translated text to your reference translation and it do this by calculating overlap of word and grams between your translation and the reference text plus some penalty like the length of the text is different. So it's very traditional methods. Method it's actually correlated well with human evaluation if you ask people like how well this translation is in terms of quality it's correlated well with blue scores. Other metrics like TR, TRF, Roush basically trying to do the same things because they try to compare this like reference text and your translation by different metrics by different overlaps but other two metrics, BERT and Comet it's like let's say new generation of metrics when they what they try to do it's they use actual multilingual models like infer first model BERT and Comet I think their own and they just build sentence embedding between your reference translation and your translation and see how close your sentence embedding are and they say close, yeah, it's good translation and they fine-tune the model specifically for this use case. And again like we see more and more in research that people start moving from blue to like BERT score and Comet, especially Comet. Okay, so yeah, what left is results. So for general comparison, yeah it's, I'm sorry it's probably hard to read because it's a lot of data. But what happens here is so this is different data sets. And for each data set, so the first column, first two columns is the cloud engines, Azure and Google. Then next three is open source engine, Sonima and Marian. And then the next three is multilingual ones. So 10 PM 50 and BERT 50, M2M and NLLB. And I know it's hard to read, I'm sorry, but basically if you look closely at these results main conclusion for us was that open source engines are actually on par in terms of quality with cloud solutions. So because if you talk with people about what translation to use, people usually tend to say, oh no, let's just go to like Google or Azure because it's probably better. Turns out not always. And like open source solutions are actually on par with them. So it was one of the important conclusion for us that we can actually use and implement open source models for an NFT inside our company instead of using cloud. Potentially another piece was around multilingual models. So if you can see, so here's a comparison when the first column is Marian that I put there for comparison and then the next one is different versions of multilingual models like big, small, and so on. So the first model from Facebook M-Bard 50, it was really bad. At least again, as I said, they have different use case, but for English to Spanish translations they were really bad if you compare with Marian. But then they improve. So M2M 100 was like a year ago, was released a year ago. And it's actually like have better quality. And then they released, like last week they released an LLB that actually on par with Marian on some data sets. Not all of them, but on some of them they're actually on par. So maybe in the future it will make sense just use multilingual models for everyone. Another problem that Marian is like 300 megabytes and an LLB is five gigabytes. And it's obviously the same difference in speed. In other tests that we run was around grid search. Again, we want to make translation faster. And for open source model, basically it's transformer model. So they have encoder, decoder, and for decoders they usually do the beam search to generate the results. Turns out that for all models there's not much difference in quality. If you just use grid search or beam search with I think beam of four. So the models are almost the same. Like if you compare them, like so here each pair is like the model is gritty or beam search. Again, what's important conclusion for us because grid search are obviously like faster placing usually two times faster. So again, easy to use. Yeah, and as I said about metrics. So this is different graph. So we have different models here on the X axis or it's a sort of different translation engines. And then each line is different scores. So like blue score, cheerf, berth, comment, root, tier. And as you see, if you just sort them by, let's say blue score. So from a smaller score to higher score, all other scores will be sorted as well. So there is no difference. So if one model is better than another by one score, they will, it will be better than another by any other score. So again, in our use case, because we compare like different models with big difference in quality. If you care about one, two blue points, you usually start looking at other metrics because then it will be important. And yeah, couple of words why we actually doing this if you will come like one of the reasons. If you compare the cost of translation. So if you want to translate one million characters, it's around 300 pages. In Azure, it will be around $10, like in Google 20, in OS 15. I don't know where gets this numbers to be honest. But if you just implement the same open source engine that we saw that have the same quality and on Azure GPU machine, it will take around, I don't know, five, 10 minutes to translate all this data. And you just pay 30 cents for this big if you're ready to support this. And yeah, all of this framework that I was talking about is actually open source. So you can go like github.com, Optum NMT and it will be there. So like if you have a need to evaluate machine translation engines for whatever reason, it's there, you can use it. And if you have a question, you can reach out to me or to Sahil here in the room. So yeah. And yeah, I was, people make me to put this here. We're actually hiring. So we have a lot of position open here in Dublin and like across the world. If you're interested, again, reach out and we can talk. Thanks. Okay, thank you. We have about six minutes for questions. If anyone in the room would like to ask a question, please come to the microphone. Yeah, thank you for your talk. You come to the conclusion that the open source models are about as good as the commercial ones, but in the graph, you do see a slight difference. So like how do you draw the conclusion that there are? Yes, so first of all, like one of the conclusion was as you see, like the quality is different between different datasets. So sometimes like one engine can be like really better. It's actually a problem in machine translation because probably like, for example, like here in Paris, you want a Nima is much better, but it's probably because they were straight on it. Because again, it's public datasets, people use them. But for our use case, we don't care much about, so for example, if you look like this dataset, Nima. So the difference between let's say like Google and Marin is around two blue points. And in the, like in real world, it means that nobody notices the difference. Again, it's important for some application, but if you're talking about let's say application when people just translate stuff, like when agent talk with customer and they want to like communicate better, they may use machine translation to get some suggestion in other language. Nobody will notice the difference into blue points. So that's why I say, again, for our use case, it's call me the same. And another piece was that we actually wanted, we want to find you on the models. For our use case, it's also possible to do in cloud, but it's much, much more expensive. With open source, it's much easier and much cheaper to do. And then you actually see the big difference. Thank you. I agree, thank you. Can you tell me something more about the licensing of the open source models? Can you use them commercially? How does it work? So it's, yeah, so it depends on the model in short. So basically the models that we used, they don't remember the license for all of them, but usually they allow commercial use if you say that you use them. But it's important to check because some of them are actually released by non-commercial. So for example, as far as I remember, Facebook models, like we included some translation just to see like how well they perform. But I'm not sure that we can use them in particular because they are probably not commercial. So it doesn't look like we have any remote questions either. So that ends the session. Once again, thank you, Anton, for the interesting talk. Thank you.


Small, big, beautiful man, yeah! Joey, you got a lot of things in your mind and you got a particularly beautiful mind. So I want to give you the opportunity to speak your beautiful mind, which is the name of your show. I shouldn't be saying this so much, but like I'm saying, I threw out four topics for you. We got masculinity, what's the deal with makeup? That's majority of sign filled impression. We have small people, big cars, that's topic number three, or the end of the world and whatever inspires you to talk about any of these things, go for it. But I'm giving you 10 minutes, right? And the goal is you don't stop talking, okay? Okay. Um... Hold on, hold on, hold on, hold on. Hold on, don't start yet. I'm wondering if I should have like a on-screen timer. I'll put an on-screen timer when I edit this. All right? All right. Okay, so you ready? Joey, speak your beautiful mind. So Ty thought of a few random topics and one of them was masculinity. And I have a few thoughts on this. I didn't meet my biological father until I was 24 years old. I didn't have a father figure until about age 13. And so there's a lot of kind of things I had to learn on my own about being a man. And I learned from my opinion that there's about as many ways to be a man as there are men. And when the whole, the trans movement became more and more public a few years ago, I was completely in support of it 100%. There's just one, there's one thing about it that has a tendency to bother me just a tad bit.  And in my 27 years of living as a cisgender man and identifying as a man and not having a problem with it, I couldn't tell you what a man is supposed to feel like. I knew from the beginning that gender roles are a social construct and that, you know, we may have brains wired a little differently here and there. But ultimately, around age 17, maybe 19, I realized that I don't have to like things that other men like. I don't have to be interested in cars, which I'm really, really not. I mean, I'd like to learn how to work on them just for, you know, out of pure interest, but it's never been like an obsession, never been that much into sports. I was, I tried out for contact sports in about eighth and ninth grade year, but it didn't work out for me. I ended up becoming the water boy. I was a theater kid and, but I've never, with all of that, you know, with all of those reasons, excuse me, I've never had a reason in my mind to question my gender and all of that. So most of the time I want to, I want, it's not really any of my business unless a trans friend of mine makes it my business. Most of the time I think that it's a legitimate situation where there's a neurological disorder that's backed completely by biology and anatomy and science and, you know, they just need to continue to see doctors and, you know, ask what the doctors would have them do. And it's just not really, I don't really, I don't hold an opinion on that part at all. And I haven't, from the beginning, like it's just, it's your body ultimately. So, but when it comes to masculinity and femininity, it's not, I don't think it's an exact science and I don't know the first thing about sociology or psychology, but from my current perspective, and I plan on learning about those things, from my current perspective, just based on living, it just seems like, what is, what are those things? What is masculinity? What is femininity? How do you define it? How do you measure it? How do you, you know, like, and who determines what that is? Who determines what's more masculine? What's more feminine? And if you have more masculine traits than feminine traits, does that mean you're a man trapped in a woman's body or vice versa? And it's just, and then I think things have just gone a little haywire. And I'm bisexual, so I identify with the LGBT plus community. And so I think things have kind of gone off the deep end where all this is concerned. Because now we have, what is it, like, 51 genders. And so it's a little, it's a little confusing to me. Because, see, if I were a trans person and I were convinced and certain based on all kinds of evidence, both anecdotal and biological, medical, all of this evidence that I am, let's say, a woman in a man's body. Like, I have a woman's brain and I know it. And I'm trying to transfer from woman to man.  We can identify with whatever we want. And it just doesn't matter anymore. And it's just, I don't know. I don't know about all that. Because it just doesn't seem necessary. And at the same time, it also just, it's not, it doesn't seem like science to me. It seems very much like astrology. And the things in the Mars Briggs personality test. And the things kind of in that vote of the non-exec scientists or almost that I dare say pseudo science, it just doesn't seem, I think it's also self-defeating and counterproductive when it comes to helping the trans community, helping the world take the trans community seriously.  We can just be whatever we want to believe and do whatever we want to do. And it's just, and I think it's a lot of giving into what's called post-modernism, which is a lot of what religious people get into too. It's basically the belief that science is just an opinion or science can't explain everything. And it's just, and facts are malleable in a sense. And it can go down that road. I'm not asserting an absolute truth. I'm not asserting with anything with 100% confidence or anything like that. It just seems to be that with all that, it can lead down a road to where facts don't really matter. And I think the facts are on a trans person's side. And so if we kind of just start throwing all this out there that we can just do whatever and identifies whatever, it's just, I don't know about that. So, but I mean, I don't have enough knowledge and expertise to write a book on it or to do a peer-reviewed journal on it. But it's just, that's just my current thoughts on the matter. I just feel that you really can't define masculinity and you really can't define femininity. And I think it can cause somebody who is just perfectly natural and perfectly normal, like a man who's interested in theater or a man who's interested in makeup to become self-conscious about who he is. That if I'm a man, I guess if I'm not interested in all these things that most men are, something must be wrong with me. And I think that can, these kind of things can cause that kind of mindset to go down that road too. So it's, and for some reason, what the next question that Tyrone asked was what's the deal with makeup? I don't know what's the deal with makeup. I had a lot of fun with it when I was in theater and I actually plan on doing drag someday just for fun. Cause there's actually a lot of cisgender males that do drag. And so I'm completely okay with it. I think thought theater was a lot of fun and I miss it. And I plan to be a part of it again someday when the world starts to turn again. So, tell you, yeah. Minute 30, keep talking, keep talking. Minute 30, I got a minute 30 seconds. Okay, considering that now that I mentioned it, another topic of what Tyrone said was the end of the world. It kind of does feel like the end of the world.    And, but the way that I keep saying is that that I just try to tell myself that there's people out there that are experts that have done this for years. I've been dealing with all this kind of stuff for decades and they know who to contact and they know who to talk to and they know what to do.  So I hope it's not the end of the world, but if it is, I'm a very adaptable animal. I mean, our species wouldn't have made it this far if we weren't adaptable. So I plan to just tap into that no matter what happens. So. You have 30 seconds left, keep talking. I have 30 seconds. Keep talking. What's up with small people in big cars? I mean, is that not, that doesn't seem very safe. All right, time, time, time, that was really good. All right, so I don't know if I should do like a summary, but you had some really good quotes in there. The ones that I wrote out were when you were talking about masculinity, you said there's as many ways to be a man as there are men. And I thought that's such a kernel of truth and such a well, that's like a book cover right there. That's really good. And then what is masculinity? How do you define it? How do you find it? How do you measure it? And I was like, that's something that I've always had in the back of my mind, but I never had anyone say that wasn't inside my head. So like hearing that, I was like, yeah, how do you measure masculinity? And if we don't have a means of measuring that, how do we know what is masculine versus what's masculine? What does it say that we don't really have a standard? Or what's healthy masculinity versus toxic masculinity? Yeah, like I would love to have like better standards for measuring that stuff. I also like the idea of, you know, like the state of the world where we tend to be motivated by not hurting people's feelings as much as we do avoiding like the hard truths or like acknowledging that being cool is a form of kindness in its own right sometimes or like kindness is cruel sometimes. But it can lead down a road where facts don't really matter. And I was like, wow, that is a well-put sentence. If you could edit it to make it to portray and emphasize that I'm not transphobic, that I'm 100% in support of the trans journey, I'm just kind of speaking on the particular topic of the whole 51 genders thing. Where did that come from and how is that helpful? So yeah, like if I were to speed through the summary, I think I agree with you 100%. I feel like I don't know what it's like to be a man despite the fact that I am a man. And because I don't know what it's like to be a woman, I have no frame of reference and having a frame of reference is so important. So that's such that when I meet someone that says, I don't feel like a woman, I feel like a man or I don't feel like a man, I feel like a woman. I'm like in my head, I'm like, how do you know which one's which? Because even if you were one. Yeah, I don't even know what a man feels like and I've been 100% behind identifying as a cisgender male for as long as I could identify. I didn't even know there was another choice. So it might be what there was and I'm fine with it. But I'll throw out this story because I feel like it's worthwhile to give you another 10 minutes and ask what it would take for you to change your mind or what it would be, how would you recognize if you're wrong? Cause it's good for you to have a frame of reference too. But the story that I got was one of a guy who loved playing guitar, hated playing guitar actually, but always wanted to play guitar but just didn't like the practice of playing guitar. Just like, I love the way how it looks. I love the guitars.  And I've gone to the guitar stores and I'm playing and it's like, this doesn't just make sense to me but I like playing guitar. And so what happened is he went over to his friend's house who had a lefty guitar and he's like, well, I'm always playing right handed. Let me just try it left handed. And when he had the left handed guitar in his hand he was like, oh my gosh, I'm left handed. And he didn't even realize it his entire life. He was just like, this just feels right. And when I did this, it just felt wrong. But when I did this, it feels right. And he played it and he was still bad. Like it wasn't like he was an amazing guitar player but it was just the idea of when he tried the other way, like when he knew the status quo of what he was started off with was not good for him. And when he tried something else or at least opened up that door it was just made a lot more sense to him. And I can understand someone who's like, steeped in the masculinity lifestyle, opening up the door to like something a little less masculine and more feminine maybe or something on the other side of the fence and being like, this feels better than where I was at right now. I may not even be a better looking woman or a more attractive male, but I feel, I don't recognize, I don't identify with whatever's on this side. I feel much better over here. And I will take some work obviously but I'm willing to make that transition. That's why I identify as trans. I was like, that makes sense. Like that, I didn't have an issue with it either way.  Cause I just get along with them way more than all these other women was like, I get that. It's like, I'm totally fine.  I'm not going to say, hey, what counts and what doesn't. I'm like, it's all good to me. And then I also, if I had my opinion, I would get rid of the terms lesbian, gay, trans, maybe keep them if you really need to be specific but lesbian, gay, trans, LGBTQ, PA plus Q, all that stuff, get rid of it. Call it pride, capital, all letters. That's a marketing move straight from black people. We, that's a gimme, that's a freebie right there. Call it pride. It's like pride folk or like pride. And I'm like, that's good. Cause you already got pride parade. There are people who are trying to make that move. And I'm like, yeah, cause that's way easier to understand. It's like, hey, we're different, but we're all different. And we can be proud about what makes us different. So we can show pride in that. I'm like, that's cool. I include everybody. 100% behind that.


Welcome to OTAN, your Outreach and Technical Assistance Network. Newseum ED is a free website belonging to Newseum, a museum in Washington, D.C. They are dedicated to the topics of news and history. The focus is on First Amendment rights and helping students develop critical thinking skills through thousands of lesson plans, videos, classes, digital artifacts, historical events, and critical debates. Newseum ED is a great way to build the critical thinking skills needed for our high school equivalency students. There are several different ways to explore Newseum ED. First, let's take a look at some resources by topics. There are six different topic areas. Understanding the First Amendment, finding reliable facts, making historical connections, examining the powers of the press, navigating a religiously diverse society, and exercising my rights. One of my favorites is finding reliable facts. Let's take a look. Inside, there are numerous tools and techniques to evaluate information. This first one is a collection of resources that can be broken up into smaller units. Let's take a look at one lesson. At the top is a description of the activity. This lesson takes approximately 30 minutes to an hour to complete and is appropriate for seventh grade through college-level students. There are step-by-step instructions and all the needed resources. At the bottom, you will also find the related standards. Going back to the homepage, I want to show you one more way to find resources. This time, I'm going to select ED tools. Here we can search by keywords, type of tool, topic, grade, and more. The content can be filtered by grade level and adapted for English language learners and basic skills students. Museum ED topics are important to all our students. You will want to get your own free account. Please be sure to check out Museum ED. It is a great resource for you and your students. Thank you. And as a reminder, OTAN supports California Adult Education Agencies who are integrating, improving, and maintaining technology use in the classroom. Always stay informed by visiting the OTAN website at www.otan.us and be sure to subscribe to our YouTube channel, like us on Facebook, and follow us on Twitter.


Usually, as you have seen, I chair only sessions with the political leaders, but this is a fascinating opportunity for me to have a discussion with a great leader in technological advancement. And Sundar Pichai, I think we have something in common, not only the interest for new technologies, and you know, you all got the book which I just published, Shapings, The Force Industrial Revolution, but we have something else in common. You cultivate a very low-key style, I try to do the same. You, just Davos, outside Davos, my wife can tell you, very low, very low key style. But your journey has really been a remarkable one. I mean, you came from relatively modest origins in India. You studied in India.  But, Sundar, you added business school. You entered Google.  Now you are at the top of a company which I have, maybe I should tell you a story. I was visiting a prime minister, I will not tell you who. And he told me there are only five real global powers around in the world. And he mentioned three countries, I will not tell you which ones. And then he mentioned Google. So that's the context. And what I would like to ask you first is stepping into the role of leading such an important, I would add, maybe also impactful company, life-changing company in many ways. You took over from the founder Larry Page, quite big shoes to fill. And since then you have announced a major conceptual shift for the company from moving from mobile first to artificial intelligence first. And change is never easy. So what have you learned since you became CEO? Well, first of all, it's very exciting to be here. And thank you for doing this. You know, when I stepped into the role two years ago, one of the things that struck me was the core mission of Google in some ways is a timeless mission. You know, we undertook this journey to organize the world's information. And if anything, today people deal with more information than ever before. So one simple way to remember it is 10 years ago we launched Gmail, which was a way we searched through text to organize users email. But today we launched Google Photos. Thanks to AI, we can now search through photos. Well, over a billion photos get uploaded to us every day. We can search it and organize it. So if you type in and search for hugs, we can find pictures of people hugging each other. So the mission hasn't changed, but we can do it radically better with the AI. In terms of how we do it, I think maybe that's been a more important realization. As Google, we want to serve everyone in the world. So it's a privilege to serve billions of users every day, with which comes a lot of responsibility. You know, we want to be a global company, but we want to be local in every country we operate in. And I think with this comes the responsibility to engage better with society. And so understanding that being open to feedback and really interacting and engaging more with the external world is a big part of the realization. Sundar, speaking about artificial intelligence, it's not just a product. It's a systems enabler and changer. And so there's a book out which is called Reapons of Mass Destruction. And she talks about how big data increases inequality and threatens at the end, democracy. And so that algorithm can perpetuate bias or injustice. How do you respond to it? You have a special responsibility. You must have a response to this question. Look, I mean, one of the things in that was I've seen, there are a lot of important questions about technology, its impact, and what role artificial intelligence plays. First of all, I think it's good we are talking about it. As humanity, the way we solve things is by getting concerned about them. So if you take climate change as a framework, we all started getting worried about it. And that's why we spend a lot of time working together, creating things like the Paris Agreement and then working on solving it. I think about things like AI the same way. AI is probably the most important thing humanity has ever worked on. And I think of it as something more profound than electricity or fire. And anytime you work with technology, you need to learn to harness the benefits and while minimizing the downsides. You know, stepping back when you think about, you know, a lot of problems in the world today, it's because we typically have a constraint on resources. AI, for the first time, I think over time, offers a different construct. Things which are constrained and look like a zero sum game today may not be so in the future. Take education, for example. It's really difficult to educate people in a cost-effective way. AI may fundamentally change that equation. It may make it possible for us to have clean, cheap, renewable energy for the future. So I think a lot of things will play out in more positive ways than people think. But the risks are important. And I think the way we solve it is we think ahead, we worry about it. We do things like from, be upfront, you know, have ethical charters, think about AI safety from day one. Be very transparent and open in how we pursue progress there. And figure out global frameworks by which we can engage. Just like Paris Agreement and Climate Change, you know, using forums like this, I think we bring people together to engage on the hard questions. And I think answers will emerge. But I think it's important to be positive about it, especially in the West. I think, you know, I still today have a wonder for technology growing up. You know, I didn't have a phone for a while, telephone. We waited five years. We got a telephone. It fundamentally changed our life. People came to our home to make phone calls. You know, I still remember the joys of technology. And I think that will be true for AI. So it's important for us to explain that and bring the world along with us. But what you describe is, let's say, AI creates or has a common good nature. But son, you have AI plays a very important role in surveillance of efforts by governments. It is a key factor also in cyber security. How can you avoid that AI at the end leads us not into a totalitarian state? You know, to me, the only way to solve some of these deeper issues is global multilateral frameworks. To me, the kind of questions you're asking needs to involve G7, G20 discussions and, you know, countries we have to agree to demilitarize AI. And I think that's a common goal. Country should work towards. You know, there's no other way to solve it. One of the good things about it is AI is kind of an equalizer. So I think over time people will realize it's tough to weaponize it because everyone will have the same ability back. And I think that gives a framework for us to think about. You know, so you need a global stand down, global consensus not to use it for military purposes. Going to be very hard. But I think, you know, is the kind of framework we all need to work towards. And that's the only way out of it, I think. Let me let me turn to big data. And I saw in an interview last year, you spoke about, and I quote you, the data belongs to the user. We are stewards of it. So it was, of course, in response to concerns about Google's extensive reach over personal and consumer data. Also, Mrs. Merkel, in her speech this afternoon, spoke about the question, who actually owns the data? Now, at the moment, it seems the data are owned by some big companies like yours. Should we trust some big tech companies? Or is there any kind of self regulation? Yeah, I mean, it's an important question. I think today we always think we operate under the framework that users use Google because they trust us. And it's something easy to lose if you're not good stewards of it. So we work hard to earn the trust every day. You know, one of the biggest concerns with data as you saw through last year is security of data. You know, so for example, when you use Gmail, we work hard to make sure we keep your email safe from all kinds of attacks that's possible. And I think that's the framework by which we operate. But I think I always think data belongs to the user. And as companies, we are only stewards of it. You know, we work hard on making data portable. Tomorrow, if you want to use some other service other than Gmail, we actually make it very easy for you to take all your data out of Gmail and go use something else. I think that's an important framework to have. AI will actually make some of this easier. I think AI will allow us to put the user more at the center and build trust systems around it too. So but I think what you're asking is got to be the foundation for it. I don't think any single company should own these things or that's not the contract we can have with the world. That's a very important statement, I would say. And not necessarily what the public always feels the public feel. So data is in the hands of the big companies and they do whatever they want to do with it. You know, I think then they won't use this again. And so, you know, it's something and increasingly when you take areas like health, if you really want to do good stuff in the world, we you know, data security and trust are the pillars of it. So we have to be very careful. I may ask you a relatively concrete question. Google has a global share of more than 80% in internet searches conducted over PCs or smartphones. And in June last year, I think it was June last year, the European Commission find you, if I'm correct, 2.4 billion dollars or euros, euros for preaching antitrust rules in the shopping search service. What are the specific responsibilities of corporations in the technology sector who have acquired a quasi monopolistic position? And of course, underlying here is also the question. How do we create a tax system, which let's say fits this new type of companies? Yeah, two questions in that.  You know, large platforms can be disruptive. So even when you put the user first, and make a decision, there are going to be winners and losers, right? And I think I think that has an has an impact on us. But from our standpoint, you know, we are pretty determined to stay focused on the user. And we're very confident, you know, products, we do benefits the users, including users in Europe. And that's the conversation we have with the European Commission as well. Tax is a is a great question, you know, as a company, you know, we pay, if you look at what the last five years, close to 20% in tax, we are happy to pay a higher amount, whatever the world agrees on is the right framework. So it's not an issue of the amount of tax we pay, as much as how you divide it amongst the various countries. The way we are approaching it, I said earlier, I want to, you know, I want Google to be a global company yet local in every country. Today, tax system works based on the R&D you create. This is why we are actually investing and adding engineers globally. I was in France on Monday, we are starting a AI research center there. We're going to be hiring a lot of engineers in France. And we're doing that in Europe. Over time, that normalizes the value of tax we pay, right, because it reflects on where you create value. So that's one good basis to do that. But we are open to any construct, we encourage the OECD to to actually solve these issues, which makes it much easier for companies to, companies to operate on. When you depend very much on the performance, resilience of the internet, and of course, there's also the philosophical or ideological issue about internet freedom. So let me ask you two or three questions. The first one is, at the recent Internet governance forum in Geneva, a number of people raised the fears that the internet, at least in the form we know it today, in 10 years will not exist anymore. What is your take on it? You know, to my earlier frame up, we always worry about things. You know, somebody forwarded me an op ed 150 years ago when bicycles were invented. And in the leading newspapers, they were worried that bicycles would give freedom to girls, and they would get on bicycles and ride very far away, and would really cause society to break down. So I think that's how we worry and constantly make things better. But things turn out very differently. To me, you can think about the internet as a set of technologies, and that will continue to evolve. But if you think about the internet as an ideal, the ideal of giving anyone in the world a chance to get connected with everyone else, give them opportunity at a global scale. So today, using Google, when there are ranchers in Idaho in the US who export their products globally, there are artists who put up things on YouTube, and they have their audience globally. That's what the internet does. I think that clock is going to march on. That ideal I'm not worried about. As a technology, if anything, I think it'll get better.  These are exciting developments. So I think the internet, we have to worry about it. But, you know, I'm optimistic. I'm optimistic about technology, not because I believe in technology, but because I believe in people and humanity. And, you know, I think it's a good construct to move forward. It's also one of the, let's say, key objectives of the World Economic Forum, always its new center for cybersecurity. But when you speak about, I mean, you have to have equal access if you really want to have a democratic system. But now the recent decision by the US administration to allow a two-speed internet, isn't that already the demise of this democratic concept? You know, net neutrality is an important concept.  The good news is if you see the public sentiment where things are, people really cherish, people don't want their traffic to be discriminated in a certain way. And so I think that's an important ideal and is giving rise to a lot of good debate. You know, I'm hopeful we cherish those principles and preserve it for the coming years. Who will actually win and who will lose if you have such a different speed internet? You know, part of the reason as a big company, we advocate for net neutrality.  But when Google was a small company and getting founded, they had a chance to reach users. And I think that's the principle we are trying to protect. And so, you know, I worry less about Google as a company when it comes to these things. But for the next set of entrepreneurs building their services and trying to reach users, and I think that's the principle we all need to fight for. Sundar Pichai, I asked you maybe a delicate question, but a question which is very much in the mind of many people. And I related to the protests in Charlottesville in Virginia. And in the aftermath of what has happened, a number of technology companies took action to remove hate groups from their platforms. So in some way, you are exercising a, how shall I say, a qualification of what's on the platform or not. And this leads to the question, should digital platforms be able to unilaterally decide what content is acceptable and not acceptable? You know, I don't think that's the right outcome. And you know, as a digital platform, we feel like we are on the cutting edge of the issues which society is grappling with at any given time, where do you draw the line of freedom of speech and so on. The way we approach these things, so for example, over the last year, we made tremendous progress in how do you deal with violent extremism. The way we did it is by engaging many NGOs, counter-terrorism organization, non-profits, and they guide us on where to draw the line, right? So that kind of gives you a framework as to how to think about it. Sometimes the government helps us by giving clear laws. And you know, sometimes that gives you a clear framework. But I don't think I think it's the function of a democratic society to decide where to draw the lines. You know, I don't think it should be in the purview of any single company or companies to do that. So we have to figure out how to engage with society, get the input, and that's how we are trying to draw the line. So it's some kind of a self-regulatory mechanism which we might. We are doing this with the European Union and developing codes of conduct around hate speech. And you know, we are doing that in a voluntary way. And I think that's that's a good. So you would feel what the German legislation which came into practice, I think, the first of January goes too far or?  They are trying to solve important problems and we want to be a constructive partner. So there are many regulations in which we work together, you know, be it GDPR or be it, you know, even right to be forgotten, et cetera. So we are always working together to find the right right place to be. And I think that's the way we approach it. When when you look at Google's histories the last months and preparing for this session, you, but I come back to the leadership issue. We touched upon at the beginning. So what's a former employee? James Stammer, I think, who wrote the memo criticising the diversity initiatives within the company. And was there anything in the memo as you as the CEO took to heart? I mean, as a leader of 70,000 plus employees, what was your reaction? Look, as a company, we support freedom of speech and we give a lot of platforms to do that. We have to understand in the context of a workplace, you know, the representation of women in tech is very, very low. Technology products affect everyone in the world. So the only way you're going to do it better is by involving more women in the development of technology. I think it's a moral imperative to do that. And so clearly we want to create an environment which is more supportive for women. And it's, you know, it's personal to me.  And, you know, I saw first hand how hard it is to function in an environment like that. So we're just trying to create an inclusive culture for all Googlers. And, you know, and that's why that's what the debate was about. I'm glad there was a public debate in which people, you know, engaged on these topics better. But so is a larger debate.  But if you look at the statistics, only 7% actually only 17% of the women's working for you, whole tech jobs. And it's a larger issue because study of the American Association of University Women shows that women in tech are actually declining and not increasing. So what are you doing to help such a technology sector continues to embrace diversity in gender races? I mean, in all aspects of social life. I mean, it's, it's an important question. We engage as a company in lot over the past years. We start a conversation in tech by being transparent about, you know, the presentation we have internally. I think there's a lot of positive news there. You know, if you look at academic computer science programs throughout, you know, in places like the US, you definitely see the tide is turning. We have to, we have to build on that momentum and you have to make sure the environment when women come into workforce, especially on very technical jobs is, is positive and supportive. And, and that's what we are working on, but it's a lot of work left to do. We just have three minutes to go. Google is, is a very, as you also mentioned, is a very international, it's a global company, but still you are perceived as an American company. What do you do in order to really gives a perception of a global company? You answer partially to this question already, but Well, you know, we've been thoughtfully doing this for the past few years. In fact, if I go back and look at the past four years, you know, almost greater than 50 percent of our hiring is outside of the United States. So part of the way we do it is by really, you know, actually hiring in a distributed way. The second part is, I think we are engaging with, you know, local governments, institutions. For example, if you look at our digital skills training, which I think, you know, we didn't talk much about, but the biggest urgent issue to do with technology is we all need to retrain the workforce. Gone are the days when you could educate yourself once and that would give you a job for the rest of your life. How do you do that at scale is something very important. And we are doing this locally in every country. We have committed to a billion dollars over the next five years. In Europe alone, we have trained three million people and we localize these programs in every country. And we plan to do a lot more of these things. And so the way we show that is by actually being a responsible local citizen in all the countries we operate in. And we are very, very committed to doing so. My last question is you are now since a relatively limited period of time as a CEO. And you were before more in a technical function in Google. How would you define if I would ask you what would be for you the three, four main complaints of which describe your leadership in Google as a CEO? I'm sorry, main attributes. Yeah, what are some main attributes? What makes you special? I know you are low key, but let me let me ask you. So so many people can afterwards also become CEO of Google. You know, I you know, I think at the end of the day, there's a lot of dimensions to the to the role. But I think it starts still with, you know, building great products. And I never lose sight of that. My happiest moments are even in my busy schedule, if I can spend an hour or two and start my day off with engineers and designers and product people thinking about the products we are building. I think I think it's something still I make sure I spend time on. And and you know, I try to build a very collaborative team. I think at our scale, you can get things done unless people work together. And so I missed a lot on a lot of time and making sure I attract the right people and create a culture by which we all work together well. So probably the two most things. And the third is I spend a lot of time on partnerships and engaging with the outside world. I think it's a really important thing we need to do well. And and which is why I'm excited to be here. And it's a great place to do that. And how do you keep life work balance? Yeah, you know, it's a good question over time. I've learned to learn to have a healthy perspective on, you know, there's a lot of things to worry about. But, you know, it keeps changing and there are good news around the corner as well. Some very optimistic by nature and I try and carve out time. So the only way I think you do it well is I am very particular about the activities which I do meet with my wife or my kids.  And, you know, that's that's one good way to do that. Thank you. So now we are just in time. And on behalf of all the participants, thank you for answering the questions in such an open, straightforward way. And we wish you for the next 30 years at Google. All the best. So thank you. Thank you so much. Thank you.


Hey Paul, this is Megan with Oxmore Toyota. I just wanted to send this quick video greeting just to personally introduce myself. I did receive your chat that you're looking for a vehicle between 6,000 and 8,000. In this email you actually find a link to all the vehicles that we currently have in that price range. When you can, go ahead and give me a call. You can reach me at<PHONE_NUMBER> and I look forward to hearing from you soon. Bye!


The recent implementation of new environmental legislations led to a change in the manufacturing of composites that has repercussions on printed wiring boards, PWB. This in turn led to alternate processing methods, lead-free soldering, which affected the required physical and chemical properties of the additives used to impart flame retardancy. This review will discuss the latest advancements in phosphorus-containing flame retardants for electrical and electronic e-applications and compare them with commercially available ones. The mechanism of degradation and flame retardancy of phosphorus flame retardants and epoxy resins will also be discussed. This article was authored by Manfred Doring, Muriel Rakitomalala, and Sebastian Wagner.