Okay, we're live. Hi everyone, thank you all for joining. My name is Kinaret, and I'm the senior research community officer, newly joined on the WMF research team. And I would like to welcome you to this month's research showcase. The research showcases are monthly convenings organized by members of our team to recognize and share recent research on our relevant to the Wikimedia projects. For those of us, for those of you joining us live, we welcome you to ask questions in the YouTube chat or on IRC. We will monitor these channels and pass questions to the speakers at the end of their presentations. We kindly ask that attendees follow the friendly space policy and the universal code of contact. Before we begin, we have a couple of community announcements to share. The first one is that registration for Wikimedia is open. If you will be joined, if you've been to Singapore, join us there. Otherwise, registration for the online is still open. There will be a research science and medicine track. Please join to learn more. The next announcement, if you want to learn a little bit more about the work that our team has done over the past six months, the research report number eight is out now. And Isaac will share the link. Thank you, Isaac. Next announcement is that the WMF research team is hiring a research manager. This role is fit for a researcher with research science or data science manager management experience. Check out the job description in the link and our final announcement for our final announcement. I'll turn it over to Hal Trayman from our colleague. Hi, everyone. I'm going to share my screen really quickly. Can you all see this? Great. So I am a senior privacy engineer with the WMF privacy engineering team and I'm proud to announce the release of eight years of differentially private page view data partitioned by country project and page. This is a longstanding request from many sides of the research community. It's a big data set. It's 263 million rows and it's growing every single day. And it encompasses 342 billion page views, including 255 million rows and 295 billion page views that were previously deemed too sensitive to release. There's a table here to compare the old data set, which is available by the page VAPI and the new data. As you can see, we're significantly more granular. There's a lower minimum release threshold. There's no limit to the maximum numbers of rows that are released in a given day. The data is partitioned by both country and project. And then finally, this is a very exciting thing, in my opinion, we were able to release about six and a half years of data, historical data that was previously deemed too risky to release. If, I think it's gonna be a great resource for research on page views in the future. If you want access to the data, you can send these links you can take a look at these links and I think Isaac will also send them in the chat as well. So yeah, excited to see what the research community does with this. Thank you so much. Thank you, Hal. And I will now turn it over to Isaac who will introduce this theme and speakers. Thanks, Kennerette. So I'm Isaac Johnson. I'm a senior research scientist at the Wikimedia Foundation. I'm glad to introduce the July 2023 research showcase. Today we'll be hearing from several researchers on the theme of improving knowledge integrity on the Wikimedia projects. I'm excited about the showcase. It's been a while since we've talked on the showcase about this very important facet of the Wikimedia ecosystem essentially how editors maintain content that represents all significant points of view as reliable as verifiable. And this showcase is also about improving knowledge integrity. So our presenters have also literally built on our understanding of knowledge integrity to design tooling to help support it. Our first presenter will be Itolkan Baigutanova, master degree student in the School of Computing at Keist in South Korea. She's been working with the team since 2021 and expects to graduate in February 2024. Very excited to have her. She'll be presenting work on the reliability of Wikipedia through the lens of references. And in particular, she will introduce her longitudinal analysis of how sentences needing citations and low quality citations have evolved over the years. And I think this is work that is particularly timely as the Wikimedia community grapples with what it means to be in this time of large language models that can generate very persuasive sounding Wikimedia content that might not actually hold up when it's actually verified. With that, I'll pass it to you Itolkan to share your screen and present. I'm sharing my screen. Can you please confirm that you can see it? Yes, thank you. Great. Thank you very much for the kind introduction. And as Isaac mentioned, I have been working with Wikimedia's research team for maybe two years now. And for this time, I've been focusing on this knowledge integrity theme. And I am gonna present it today to you. As a master's student, it's a great honor for me to work with the foundation and present my research here today. I will introduce our paper that has been published this year in the dub, dub, dub proceeding. And the title of my talk would be Assessment of Reference Quality on Wikimedia. So let me start by stating the core content policies of Wikipedia, which you may already know. So the first one is the verifiability policy, which ensures that readers of Wikipedia can check that information comes from a reliable source. The other one is neutral point of view. Here it basically means that the information presented is unbiased and all the significant views that have been published by reliable sources on the topic are considered. And the final one is no original research, which means that Wikipedia doesn't contain original research and any material for which reliable and published sources like this should be used. And all of these three content policies basically highlight the importance of reliable sources. However, unfortunately, and as we know, not all the articles would follow these core content policies perfectly. So there can be some articles that have, that site where the information is coming from, like for example, here, but in some cases, the information may not be supported by any reference. And commonly this kind of parts are marked by a citation need of site, but even in those parts that are cited, the reference can be either reliable or unreliable. So for example, Oxford dictionary could be considered as a reliable source, but when some information is cited from social media, that may not be as reliable. And definitely it affects the readers of Encyclopedia who directly get to see this information either on Wikipedia or on the search engines. But from a more global perspective, it does affect far beyond Wikipedia. And one of the reasons is that Wikipedia, along with all this information can be used and is commonly used for various NLP tasks for creation of knowledge graphs and training of today large language models. And recognizing this importance of referencing on Wikipedia. In our research, the question that we wanted to answer was mainly how can we measure the verifiability or the reliability of Wikipedia articles through their references? And to do so, we operationalize the notion of reference quality first. And in our work, we define two metrics to measure reference quality. So the first one is reference need index, which represents the percentage of citation missing sentences. And the formula is pretty straight forward. I will not explain it in too much detail, but we do release a tool that can be used to compute this score. And the details can be found in our paper. So for the second index, which is reference risk, it is the proportion of non-authoritative sources among all the references in a given revision. And for the second index, we refer to the perennial sources list, which is a Wikipedia community maintained list of sources whose reliability and use are discussed. So basically, Wikipedia community discusses the reliability of sources and collectively decides reliability label. This perennial sources list on Wikipedia has five categories, which are listed on this slide. And the categories of interest in our research were the lowest two category, namely blacklisted and deprecated as we refer to them as non-authoritative sources. Or in other words, we refer to blacklisted and deprecated sources as risky for use on Wikipedia articles. And let me give you an example of how the two scores are calculated so that to make sure that you understand what the two mean and how they can be computed. So imagine we have an article that has three sentences as here, A, B and C. And then to calculate our first score, which is reference need, we look at the three sentences and suppose all of them need a citation. However, out of three sentences that need to be cited only that needs to be supported by a citation, only two are supported. It means that one out of three sentences need reference. Now for the second index, which is reference risk, we only consider the references that exist in that article. So in our case, we have only sentences B and C that have citations. Then we combine this with a list of reliability indices. In our case, it's a perennial source assist. And then we find that one of the two citations that were used are risky, which result in one of the two of reference risk. So I hope now you understand the two metrics better. And with this, we created two datasets. The first is 20k randomly sampled pages and the second was 10k most viewed pages. And we track revision history for this 30,000 pages over 10 year period. So from 2010 to 2020. And here we are given the size of the revisions and for each revision, we also provide the reference qualities course. For those who are interested, the datasets are also released. So you can familiarize yourself as well. So with this set up, the first direction, research direction that we wanted to pursue was to assess the reliability status of Wikipedia through the lens of its references. Or in other words, reference quality of Wikipedia. And to do that, we looked at the evolution of reference quality over the 10 year long period. So here, the first score I will show for reference need. And we can see that reference need has steadily decreased throughout the analyzed period. And in year 2020, a mean reference need score reached nearly 30, 38% for the two dataset that we considered. What's also interesting is to observe the evolution for topics. Here we considered this four meta categories, history, STEM, geography and culture. And we can see that the culture-related articles, which are dashed dotted line, are shown in dashed dotted line, tended to have better citation coverage. And better citation coverage means a lower reference need scores. And for our second metric, which is reference risk, we observed that throughout this analyzed period, it remained below 1% for both datasets. And what's also was interesting is that it decreased here for both datasets after 2018 when the perennial sources list that I introduced earlier was created, which is the reliability index list. Regarding the topical distribution, we can see that STEM-related articles, which are shown in dashed line, tended to have fewer risky references in them. So we showed the distribution for the higher level categories, but we also wanted to show the distribution of reference quality for a more fine-grained topics. And consistent with what we saw in the evolution plots, we see that STEM domains like biology, earth libraries, they tended to have lower reference risk. While they also had tended to have higher reference need, like for example, mathematics and physics and also linguistics, they are in the more like right side of this plot. On the other hand, for culture-related domains like media and biography, they tended to have higher reference risk. And what higher reference risk means is that they tended to have, that it tended to cite unreliable references more often than other topics. Now, once we assess the status of reference quality, we also wanted to suggest the strategies or assess the factors that affect a reference quality of articles on Wikipedia. And as Wikipedia editors are the major part of the content and as they are the ones who create the content and maintain the content, the first thing that we did, we looked at reference quality and editors. And what we did, we conducted quasi-experimental analysis using matching techniques, where we looked at experienced and novice editors and how they changed reference quality scores. So we have these two scores and DRN and DRR basically means the data or the change in reference need and reference risk scores. So we can see that for the experienced users, on every issue they use, they tended to reduce the reference need and reference risk scores, meaning that they improved reference quality. And for novice users, they tended to increase or not, or almost not change the reference quality scores. So they had the positive change of reference need and reference risk. Another experiment that we conducted was to look at the interaction of editors between each other and especially novice editors with expert editors. And in our experiment, the scenario was co-editing articles with expert users. So we can see then there are two groups of users, those who co-edited articles with experts and those who did not co-edit articles. And then we observed that novice users who co-edited articles with experts were more likely to avoid increasing DRN and DRR scores, meaning that the change, the increase in need and risk that they caused was lower than for those who didn't co-edit with experts. We also looked at the importance of community initiatives like perennial sources list and how it correlates with reference quality. And here we look at the lifespan of risky sources one year before and after they get classified as bad or unreliable in the perennial sources list. And in our experiment, we define lifespan as the time elapsed between the addition and removal of reference, not to complicate the speaker too much. I will just explain the median lifespan, given the example with the median lifespan. And we can see that the median lifespan of risky references for before they get classified as bad was about 150 days while for risky references after they get classified as bad the lifespan decreased to around 45 days, which is more than three times lower lifespan, which may indicate that once users, once editors are aware of sources being unreliable, being unreliable, they tend to avoid or remove such references. Now, I also want to just briefly mention the further extensions of this work because our work was limited to English Wikipedia, but we recognize the importance of multiple language editions of Wikipedia. So we also wanted to extend and understand the reference reliability status across multiple language editions of Wikipedia. So we use the same classification for sources into unreliable and reliable as the one that I introduced earlier. And we looked at the distribution of reference reliability across here I show for 40 editions with the largest number of articles. And here this gray area shows a 90% confidence interval. What I think is more interesting to observe here are the outliers at the bottom and above the confidence interval. And for the outlier below the confidence interval, we can see English Wikipedia, which means that it has a larger proportion of articles citing reliable sources relatively to other editions. And this result is consistent with what I introduced earlier because editors being familiar with this classification into reliable and unreliable, they try to remove or avoid using such sources. On the other side of this distribution are here, I list three, Russian, Armenian, Chinese, which are here, which tend to have a bit larger proportion of unreliable sources compared to other editions. And we were curious in why this happens and we qualitatively assessed and checked what kind of sources are causing this kind of trend. And when we check what kind of sources were causing this kind of pattern, we could observe that for Russian and Armenian, it was caused by a Russian newspaper website and for Chinese Wikipedia, it was caused by a Chinese tabloid. And these are culture and country-specific information providers for those editions where for Russian and Chinese Wikipedia's, the Russian newspaper that I mentioned and the Chinese tabloid, they were not yet classified as bad. To sum up, our work shows that the reference quality of Wikipedia is improving, which is a positive sign. And we also want to emphasize the importance of expertise sharing. Maybe the platform should consider matching editors of different experience so that they can co-edit articles together and collaboratively create the content. Also, we want to highlight the importance of community efforts like creation of perennial sources list that can greatly affect and help with maintaining the quality of the content. Also, we hope that our work inspires more active discussions within non-English communities and the potential for creation of global reliability index, especially aiding this smaller edition of smaller communities. And these are the links to our data set and our paper. Thank you for listening for the presentation and here is my contact information in case you have any questions. I am open to questions now. Yeah, thank you, I talk and it was an excellent presentation. I'm going to start with a question from YouTube, but then I'll open up the other folks in the room and any other questions we have coming in from YouTube or IRC. The first question is how you might evaluate sources that aren't on the perennial sources list for reliability. Yeah, this is one of the concerns that we also had. So we tried using external reliability indices in our work as well. I didn't present it in today's presentation but in our full paper, you can have a look. So we assessed some of the popular external reliability indices, but what we found is that external reliability indices might not be as appropriate, not as appropriate, but they may not cover as well as locally maintained lists because the sources used in Wikipedia, more specific for Wikipedia were weather covered by the perennial sources list. That's why it was our choice for this work. Yeah, that makes a lot of sense. I also want to call on Pablo to put you on the spot here because I know you've been thinking as well about this, how do we evaluate sources that maybe we don't have these community annotations for explicitly? Yeah, well, in some cases, after bringing that, this is definitely an interesting topic. Thank you, I'm talking first in this work. This is really relevant because it's so the impact of these community efforts and also like limitations about how limited is this list and then the need to extend that resource. So there are some words that is based on this moment to try to quantify the controversy of sources inferred by data to try not to create another list, but maybe to have like an alternative list to assist editors in deciding whether a source is reliable or not or consensus. And from that, I have a question probably talking like when you did the analysis of the impact you focus on novice and experienced editors. And I'm wondering if you also examine like both edits, like whether like both might be in using directly or indirectly the parental source list or might be in effective the removal of risky references. Yeah, for our analysis, we excluded both edits. So we didn't consider both users, just the human editors. Because, and also we excluded anonymous users as well because we cannot judge their experience. And here, the experience that we refer to is number of previous edits. So for anonymous users, we cannot know how many edits they had made before. And yes, and we exclude bots as well. I'm gonna, Papa, I'm actually gonna forward on a question from YouTube that was questioning or asking whether you know bots that do add sources just as like an additional piece of context for that question. Sir, I didn't, I didn't catch that. Could you please repeat me, sir? Sorry, that was for Pablo. There was a question about whether there are bots that do add sources, whether you know more details about that. So I thought was interesting context for the question. The add sources? Yes. Yes, actually there are a lot of bots. And what we found, we actually also looked at bot edits as well. So we found that bots, there were a lot of bots that were adding references. So I remember, so there were bots, something like the bots that were cleaning up references. So there were a lot of edits from them, from those bots that were editing the link in some way also many edits were coming from bots who like set the security protocol of the link, something like changing HTTP to HTTPS, something like that. Yeah, and Diego like mentioned in the chat, internet archive bots as well. I think there were many edits from the bots. And thank you for the question. Thanks for those additional details. I'm gonna pass it now to Tanya who had a question about the editor experience analysis you did. Hey there, thank you so much for this fascinating presentation. Just kind of on the topic exactly of this slide, I was wondering if you found differences between this experiment on the topics. So like culture and et cetera, whether you're looking into that. You mean topics and editors like for this experiment? Yeah, whether there were differences specific to topic based on this. Like, so this experiment, the way we conducted this experiment is we tried to match revisions, like tried to match similar revisions, but from the only thing that would differ is that one revision is from expert user and the other one is from novice users. And also one of the variables that we use for matching was topics. So we matched editors editing basically the same topic article, something like that. But I haven't looked like specifically what topics caused more difference in the score, but I think it is an interesting thing to look at. Thank you for the question. A token, you had a really nice slide about bringing this to more languages. I know that's something that's gonna come up in the next presentations as well. And I think with references in particular, just preface by saying it's a lot of work to scale this research to more languages because despite like being very central to Wikipedia, they're often unstructured and it makes scaling and being able to identify templates and all the things very difficult. So I was wondering if you could just talk a little bit more about the work you did there and what challenges maybe you're facing in doing some of this language scaling work. Yeah, I'm glad that you brought up this because I should have also mentioned that because what we, so we started with this question like, oh, we need to consider more editions, not just English. And then we wanted to look at the, again, we needed to have some reliability labeling for this. But then when we started to look at the reliability indices, most of them, the external ones that we looked at, they were very limited in the geographical coverage. So they would be specific to certain countries, like more like Western sources. And we even look into the perennial sources list. What we discovered is that perennial sources list, the one that we used was from the English Wikipedia, but it exists in other editions as well. However, other editions were not actively maintained or they had the very small lists and not like with explicit labels. So there were a lot of kind of challenges there that we couldn't find this kind of reliability labeling in other Wikipedia editions unfortunately as accessible as in English Wikipedia. So I think this is one of the things that I really hope that our work or these presentations can inspire Wikipedia editors and non-English communities, especially to raise this kind of discussions on reliability, yes. No, thank you. And I think, yeah, the tooling can also help by demonstrating that there is a pathway for if you maintain this content, there's ways to make it readily accessible to editors and things like that. With that, I think we can pass to our next presentation. We're gonna have Diego Sides-Trumper and Pablo Aragon or a research scientist and my colleagues at the Wikipedia Foundation and they'll be presenting their research on multilingual approaches to support knowledge integrity on Wikipedia. So in particular, they're gonna present a novel machine learning system aimed at assisting the Wikimedia communities and addressing vandalism, that'll be Diego's work and then a dashboard that relies on that system to monitor high-risk content in hundreds of Wikipedia languages and that will be Pablo. With that, Diego, the floor is yours. Thank you very much, Isaac. Let me share my screen. In the meantime, let me congratulate Aitalki for the amazing presentation. She started working with us when she was an undergrad student and now she's finishing her master and it's amazing to see all the research that you have done in this time. So thank you very much for talking. Let me start now with this presentation. So thank you very much, Isaac, for the introduction and thank you to all of you for attending this showcase. As Isaac mentioned, today with my colleague, Pablo Aragon, we will be talking about our work on multilingual approaches to support knowledge integrity in Wikipedia. So in the last years, the Wikimedia Foundation Research Team has been working on different aspects of knowledge integrity and from different approaches. So we have wrote several papers explaining the role of Wikipedia in the information ecosystem, articles, and we have also released a lot of data sets that can be used within and outside of Wikipedia to train machine learning models related to content integrity and content quality. We have also produced several insights, some of them shared as scientific papers like the one that Aitalki just presented. And our more recent work is on creating a new generation of machine learning models to support Wikipedia and Wikidata controllers. I would cover a portion of this in a minute. But we are not alone on this path. Let me highlight some of the collaborations that we are having on this. So we are having external collaborations with external researchers like people from KAIST. And I don't want to start the big list because I will miss some people, but a lot of external collaboration to develop the scientific work, but also internal collaboration within the Wikimedia Foundation. I want to mention and highlight these two teams that we are closely collaborating with. One is the Trust and Safety Antidis Information Team. We collaborate with them. Basically, we give them some quantitative insights and the outputs of our machine learning models that they can use for doing their more quality of analysis that we receive back and that help us to improve also our work and in general, the knowledge integrity in Wikipedia. And also with the machine learning platform team that we have been working on creating a scalable and high-performing APIs that can support especially controllers on detection revisions that requires human attention. And this is what I'm briefly going to introduce today. So River Risk. River Risk, it's a family of models that we're designing to automatically identify revisions that might be reverted. So meaning that these revisions that have some issues, some content issues or problems and that might should be reverted. Why we are doing this? We are doing this as you might know and I think I don't need to highlight that much here, the motivation, but Wikipedia content, the content integrity in Wikipedia is important not only for the readers of the online encyclopedia, but also because Wikipedia is powering other sites and a lot of the information that you see online comes from Wikipedia. However, patrolling or keeping this content with high quality is difficult, especially for the amount of revisions that we receive per day. So currently just considering Wikipedia projects, we receive around 16 revisions per second that is almost 1,000 revisions per minute. So we need models that helps to identify bad edits and help to revert them. And now maybe you're thinking, yes, but we already have models for that and we have systems even in the foundation like AORUS, that by the way, AORUS is a project that we have learned a lot from and all these learnings we are trying to apply in this work. However, there are several limitations with these current models. Probably the main one is that they are language dependent. So that means that you need one model per language every time that you need to train it or you want to add a new language, you need to train a new model. But especially this is difficult because now we just covered a portion of the Wikipedia language edition. So language coverage is one of the current problems. Another problem that we have with the current approaches is that maybe rely on manually annotated data for training and this is difficult. So it's difficult to keep this updated and also to grow to other languages. And probably equally important is that these models are already a bit old and nowadays we have a lot of advancing technology, especially for processing text. And maybe most of you know these large language models based on transformers that are so popular today and for sure they can help us on the task of detecting bad quality edits. However, these super cool models that now everyone is talking about, it have some problems. One of them is that they require a lot of computational resources. So you need GPUs for training, you need a lot of resources also for inference. So they are expensive in terms of resources. But at the same time they have some problems in terms of the language that they cover. There's a lot of studies that shows that these models works good in the most popular language on the internet but the performance decrease on under resourced languages. And there's several languages that are just not covered. The other problem that we have is that, okay, with these large language models we can understand the content and we can do complex things in these more popular languages but we cannot apply them in many of the other languages. The other approach that we can take is go with metadata and with some rules like what the old models are doing but the problem with metadata is that this introduced a lot of bias especially related with the editor's characteristic. And we have found that it's especially difficult for these models to perform good with anonymous or IP edits. So in a nutshell, these new models are cool but they are expensive, they are a bit slow but the problem is that if we don't use it we cannot cover it for complex edits. So what is our solution for this? Basically what we did was to develop two models. One that is pure language agnostic based mainly on metadata and another model that is based on multilingual birds a popular large language models that can understand text more deeply. The language agnostic model as I said is fast covers all languages. So now with this model that is already in production we are supporting all Wikipedia projects. If a new language is added we don't need to retrain the model the model would be covered and because it's not a language event. However, the problem of this is that as I mentioned, this misses some context and it's not super good on evaluating more complex content problems. On the other hand, as I said the multilingual model that is based on bird it's using these advanced techniques. We have put special emphasis that the model is fair on IP edits on anonymous edits. So our model performs, I will show some results perform very good for anonymous edits but the limitation is the ones that I mentioned nowadays we are covering just 47 languages that is still much larger than the existing models. We are improving in 60% the amount of language covered but this is still heavy on computational resources. So our current recommendation if you want to use these models if you're a developer or if you are a researcher and you want to use this model our current recommendation is that you use the multilingual model based on bird just for anonymous edits in the 47 language that it covers and then you use the language agnostic model for all other revisions. In both cases they are performing better than previous models and so you can use it them confidently. Let me know very briefly like in two slides explain how the multilingual model works if you want to learn more I will send you then some pointers but conceptually the model of this multilingual model it's pretty simple. The technology is complex but conceptually the model architecture is pretty simple. First what we do is that we use the Wikimedia-Wiki edit types library to identify content that has been added, moved or removed. So basically we take the diff but we not only take the diff but we know for example when some content was moved when some content was added and when some content was removed. This by the way this library the media wiki edit types is a super cool library that was also developed within our research team our project lead by our colleague Isaac Johnson that is presenting today and with Jesse that was an outreach and then contractor intern and when using this library to extract the content and then pass it and we pass in different buckets the content that was added the content that was moved and the content that was removed to this intelligent multilingual bird model and this model outputs a prediction and what we do is then we take that prediction and we mix that prediction with the metadata that we have about the pages and about the users and with this we output a prediction. Here you have some numbers and on the results you have the comparisons across different setups of the multilingual model what you see on green and you can see that I performed the baselines in all the cases what you see on your left is how the model performs for all the revisions all the revisions means anonymous edits plus a registered user edit and what you see in your right is just a result for the anonymous edits and you can see that the difference in this case is much bigger and this is very important because there has been a lot of discussions about IP edits and most of the Wikimedia communities has decided that they want to keep IP edits so having a fair system with them it's very important because what you see here as a rule base it would be a model that just rejects all IP edits and that would be not very good and we are doing much better than that if you want to you don't want to just believe these numbers if you want to test and play with the models you can go the model is already published in the lift wing documentation AP page lift wing is the system that was developed by the machine learning platform team to serve machine learning models so if you go to that URL that will be shared in the chat too you can go play with the models it's pretty simple to work with them here you have an example of how they work you can work from your command line or from a Python notebook test it here if you remember the fair example that I show that was in the slide about Mariah Carey and why this revision needs to be reverted you can test how the language is an optimistic model how the multilingual model works with it and you will get a prediction it's important to highlight the prediction the true means that the revision should be reverted and here you also have the probability so this is a highly probable bad edit also if you want to go deeper here you can learn more about this so now all the models that are hosted in lift wing all the material learning models that the Wikimedia Foundation will be serving needs to have a model card so you can have all the details about the model and you not only can read about it you can use and following our open source policy you can get the model from the repository you can check it, you can clone it, you can improve it so following all the open source path and also if you want to understand more the scientific details of how we built in this case the multilingual model you can read our paper that was just accepted and will be published on the KDD conference now in August and I will put the acknowledgments at the end so with this I will pass the microphone to my colleague Pablo Ragon that will explain what other things we are doing apart from supporting patrollers with this model so Pablo the mic is yours thank you Dave okay so we present now the Knowledge Interity Research Laboratory next slide please well so well for some time Wikimedia was considered a non-reliable source a non-reliable source for instance in education like this perception is changing in recent years like analysis like the one on the left about the last U.S. election concluded that in comparison to social media Wikimedia was refused from big tech disinformation like core content policies like prefability and human centered approaches to content moderation as represented they were found key to prevent misinformation to a great extent and also as Diego mentioned this not only affects Wikimedia readers like this piece on this story on the right published yesterday on the New York Times explained like in the era of AI and large language model like Wikimedia is probably the most important single source for turning those models so the integrity of knowledge in Wikimedia is key for the reliability of the online information ecosystem next slide please however when referring to Wikimedia most people think about English Wikipedia and there are more than 300 language editions this might be known by everyone here in the room and also by most viewers of this showcase but I should emphasize that reliability across Wikis might be very different so the story on the left was published some years ago when I joined the foundation around the time in Slate and it covered different situations of the Japanese Wikipedia that might have favored the emergence of structural and wider misinformation and disinformation in particular related to historical revisionist and also similar patterns of historical revisionist and disinformation were found in an evaluation of the creation of Wikipedia conducted by an external expert on the subject matter that was published by the Foundation for Transparency that was a report that also suggested different factors that might have influenced the emergence of these patterns next slide please so we created a taxonomy of knowledge integrity risk from our iterative review that I presented in the December showcase and then we built a system with a dashboard with sewing metrics from these categories that were computed for each language edition in each month and to accelerate the deployment and maintenance of the dashboard we relied on the superset infrastructure of the foundation that requires a Wikimedia developer account with the staff or NDA access and also because some metrics they rely on non-public data like geographical edits and view so in that regard like there is also the work served by HAL in the start of this showcase on differential privacy might bring new opportunities for that next slide so to showcase this first version of the knowledge integrity research laboratory I will examine some claims about the Japanese Wikipedia extracted from that story published in the slate so it was said that the English Wikipedia is viewed by people across the globe but other language versions such as the Japanese are primarily viewed by and edited by people of one particular nation this is true and actually the Japanese Wikipedia is the most in equal among the largest wikis by computing the genicoefficient over the distribution of views by country and edited by country it is highly concentrated by most of them in just one country and drastically for views next slide please it is also said that the Japanese Wikipedia has the lowest number of admins per active users across all language editions and the metrics comparing these numbers like active editors and admins considering sysops and brokerage validates the claim next slide please but the claim is also followed by another claim that says that a few dozen people within the Japanese Wikipedia have power over what goes on in the platform so examining power over what goes on the platform cannot be directly measured with language diagnostic quantitative approaches that is the one on the observatory but that we can observe among the largest wikis like the Japanese one has the pull of of admins remarkably older older in terms of the time since they've seen up in the wiki and that is that editors with less than 10 years in the wiki hardly become admins like they are mostly odd liars in those cases so there is a lack of concentration of editors from first generation that remain with admin levels next slide so well this was the first version of the list of sabatoids that allow us to provide data to trust and safety about conditions but my favorite emergence of knowledge and duty issues but several limitations exist for sure and there are two that were very relevant the first one is like this design is not based on reads that are informed from retrieved from a literary review but they are not part of a formal or systematic approach to know when a project is under a knowledge integrity list so there is a need for acronyms and second there are many metrics and that brings complexity when examining if a language edition is in a risky status so in alignment with the occult ratio like good models are parsimonious like they they should have come please there's a level of explanation with the less number of prior was as possible next slide yes so to address these two limitations there is a second approach the resource observatory that was implemented first as we need a ground truth we approximate the ground truth it's not a ground truth but it's an approximation of the ground truth of a knowledge integrity risk by using the reverberates model to quantify the presence and the moderation of high risk revisions and this is basically the intuition that we expect that common forms of knowledge integrity threats risk for instance vandalism can be captured through revisions with high predicted likelihood of becoming reverted using the machine learning model I might know that when I mentioned the machine learning model and for the resource observatory we're using the language agnostic one since we need to provide support for as many languages as possible and as you mentioned like multilingual one has several advantages but it's limited to the languages existing multilingual and then for the second limitation to simplify or pull up metrics we set three single metrics like the ratio of reverted revisions the ratio of high risk revisions and the ratio of high risk revision that gets reverted so this create a question for us that needs to be solved and what is a high risk revision like given a revision and score when we decide that that revision is high risk or not high or not present in a risk next slide please well so we compile a dataset of revisions in all wikis in the first three quarters of 2022 actually it's a dataset very similar to the one that was being used for training and testing the model and for 100 results that is from 0 0.01 0.02 until 1 we examine the probability distributions of revisions getting reverted and not reverted and as you can see here the results so that the model works quite well in identifying where a revision will be reverted or not in large wikis next slide please but obviously these classes reverted are not balanced they are fortunately there are many more revisions that are not reverted that are bringing value that are part of knowledge creation so also there is a non unmeasured bias towards IP edits and the model was not trained with bot edits since this bot edits are generally expected to be done in good faith so therefore we conclude that a high risk revision must be a revision done by a register editor that is not on bot and with a revered risk score it means that the output of the model upper to the threshold in which the model accuracy is maximum for that wiki that means that this is an example of the first Wikipedia and that a high risk revision is a revision from a regular editor register and on bot with a score from the machine learning model upper to 0.96 which is the threshold in which we are able to is the model accuracy is maximum like we capture like larger percentage of true positives and true negatives next slide please so yeah with these thresholds an extended data set of all revisions in all wikis in 2022 but also in 2021 we have generated a new prototype with a second version of the Norris Integrity Resource Seventory again to accelerate the deployment process we are relying on superset next step and I will show you an example with the Spanish Wikipedia because I'm taking this one because my mother language so I can provide you with a better understanding like well we have said that there is a decade trend on the rate of revisions getting reverted over time but interestingly we only focus not on the the revision rate the revert rate but the revert rate of high risk revisions we found a remarkable drop in the months of July 2021 and July 2022 that means like in those months there were like lower rates to moderate those revisions and we're interested in gathering more data to check if this is a seasonal effect maybe for instance the Spanish Wikipedia there are many editors that are visiting Spain and this is typically time for summer holidays but to give a better understanding of one of these pigs we have examined pages that presented many high risk revisions in that months in July 2022 next slide please and as you can see many of them are related to football teams and players we inspected a sample of high risk revisions in these pages and many of them there were single acts of pandalines like adding offensive nicknames to football players however not all pages are about the sports there are in these top 10 ranking two pages about politics one about the biography of Raul Castro the former president of Cuba and the page of paramilitaries in Colombia we inspected high risk revisions with the observatory this data is also presented in the dashboard and this is one example this is a revision of this information on the biography of Raul Castro stating that he was married with a commander of the Cuban revolution that is not true and this could be an anecdotal case of pandalines but if anyone examined the contributions of the editor like similar edits with this information were made in pages about the recent history of Cuba in Spanish Wikipedia but also in Wikidata and Coma so now we have one dashboard that allows to navigate very easily and very quickly all this data next slide so what's next next slide please so one future line would be to to extend this approach of using implicit annotation like repairs in other cases that are relevant for for knowledge integrity like new page creation creation article protection and also there is a model that Diego and the team is working on on Wikidata is in evaluation phase and we're interested in creating new models but also we're also committed to analyze the already existing models to continuously improve them so we're interested in examining whether like the way that these models are trained using like general dumps with data from all of Wikis are all we're representing like patterns of large than with Wikis and we should balance to get more patterns and specificities of some of every individual project also Diego mentioned like there is a bias against certain edits like IP edits and we are interested in also examining other sensitive groups of community members like newcomers and that relates to what I talk in percent so trying to mitigate like these models such on like good faith editors that they are just missing some knowledge to to to contribute to Wikipedia and last but not least like these models are using implicit annotation like repairs but we can also incorporate explicit annotations like feedback from editors to retrain the model or also the model would be retrained by monitoring data drifts that examine whether like a model needs to be retrain with more recent data that capture like new dynamics next slide please so yeah like these machine learning models that Diego has been leading this world were built with a team with Muniza Aslam we call out Okimobit and Ico too with a great support from the machine learning platform and the operation with the resources about to have benefited from the very valuable feedback from the trust and safety this information team so thank you all thank you all for all this contribution like we are always committed to external collaboration but these internal collaborations are as as important the start of once next slide and thank you everyone here today for your attention and now we are we are pleased to start the Q&A thank you very much Diego and Pablo as we kick off the questioning I'm going to pass it on to Martin in the room and remind folks on YouTube if you have questions please put them in the chat there I thank you for the presentation very interesting I have a question around the observatory and the potential scores or risks you could surface and whether I'm interested whether there's a way you have in mind I imagine it's very difficult to validate the output of this model in terms of okay maybe there's it mentions there's a risk for a specific wiki at a specific point in time is that something we could yeah validate if that is true either like maybe with specific events from the past where we know there were specific issues or so I don't know if you have been thinking along these lines thank you Martin like this is an excellent question and this has been a question that has been like pushing me for months or even years so this is one of the main side like how to retrieve like quality high quality ground through data that can help us to validate and this was not the case for for the first version and it's formally it's not even the case for the second one I'm not this is an approximation so we're trying to to work with this information specialist that they are aware of possible situations since they are in quantitative committee trying to check whether these metrics are useful to capture some risk like some references are about like committee capacity like tools like there are very effective tools in noise integrity like a news filter so how the news filter activity in terms editing the rules the hits on the rules might help to find issues or some moment with also with some external collaborators who were discussing whether using the logs of the global logs on the local blocks as a comparison on whether like a community is able to moderate issues have internally or there is a need for having like a global moderation to solve those problems so yeah this is this is one of the main the main things we have experiencing I mean I didn't into the taxonomy in the taxonomy there were like reasons that there were external and in that case we were using like external data like for the geopolitical we were using the index from reporters without frontiers and using then the geographical data to see the community according to the views and the days were coming from countries that were under risk of press freedom and using them as approximately still then is this cannot formally be considered ground truth data so any contribution that I would be pleased to to discuss because there is this working progress yeah I want to add something about ground truth in general I think this is so two things one is in terms of improving the models the material learning models one thing that we are doing is we are collaborating with community and especially now there's a project by the moderation tools team that wants to start using these models in of large scales in all wikis and one thing that Pablo mentioned at the end is that we want to give the ability the model to be retrained considering a specific input explicit feedback from admins and I think that's that's very important because this will be always an iterative process that is not only an an iterative process is kind of an adversarial problem because if there are actors that want to introduce this information if they're because one thing is like the vandalism in one side but even with the vandalism people would learn what the model is able to capture so there will be all all the time will be some drift so one thing that for us is very important is the ability of be monitoring the quality of the system and being able to retrain the model periodically and this is something that we are that we we are studying this year and I hope that we get the resources to do it and also the and also that we develop a system that can take in consideration admins feedback but also don't over fit under feedback because in the much learning space that that's the problem that you have if you want to give a lot of weight to few inputs so if we're analyzing one thousand edits per minute and we'll receive I don't know 20 feedbacks or revisions with feedback from admins per day it would be it's there's a challenge there on how to consider this and this is something that we are doing research about this this hopefully this year thanks Diego I wanted to also follow up on your working connecting it with what I Tolkien had shared and I'm wondering whether you see value if there is like possibility for incorporating and some of that work around reference risk and just you know perennial sources that sort of thing into your modeling and I guess more generally where you see the most room for improvement in detecting vandalism like what sorts of features are we thinking about sorry I can say that again to Diego the thinking about how can others not hear me all right I think Diego got frozen for a while yes sorry now the brakes but now now it's good okay I'll just repeat which is thinking about how we can connect I Tolkien's work to what you were doing with vandalism detection incorporating and features like that or just features about other aspects of content yeah that's a super good question we are in this learning process because given that this applied research we always need to see what what things can be applied on the wide later so it's we have seen that the even these models like Bert that is not the last generation like it's very far from these super large language models like like the GPT or this new Yama or there's a lot of them now is that okay we can be able to isolate some specific type of behaviors like for example adding unreliable sources or I don't know adding there's a lot of interest to patrol on images and for sure we can build models that understand images there's a lot of work on on computer vision we have an expert in computer vision the team that is medium ready so I'm sure that we have the capacity the scientific capacity to do this the thing is how we how we how close we are to being able to apply this to evaluate every revision that happens on Wikipedia but yeah and I said you you're aware that the word that the edit team is doing also on on revisions so maybe we are able to understand some specific taxonomy on the revisions and that's why the edit type is super useful but also what the edit team is doing trying to work for example with with reference if we know that this is a specific edit on a reference probably using a tool or something on the visual editor that tell us just go analyze this specific piece of text I think it's something that that we could consider so I think scientifically the problem is not that impossible to solve it's not it's not easy for sure but technically how we do these large scales I think that that is the most challenging piece yeah I'm not sure what I'm going to mention relates to what Diego just just mentioned but just just to be sure like I see one connection between what was presented about predict like predicting the likelihood of variation getting reverted or or all this work of the value of this committee initiative like pre-assurance leads like the analysis that has been done so far and many use cases are about content moderation when the content is published but maybe there is an opportunity to provide input to two editors before publishing the edit and this I make this connection because we have this conversation with Isaac and maybe this could assist editors while creating an edit like knowing not only if it's like a revision will be reverted but also an important part of this model is also it brings some and I think the API brings brings also some context about the explainability of the models explaining if there is a likelihood of getting reverted what are the reasons and that can help to inform users so what are the these conditions of that specific edit that might be then problematic in the future so they can learn before the the edit being published and is I make a connection with what I told you presented because if user is starting a source that there is already a consensus that is unreliable within the community it might be relevant to inform the user before getting that that edit and I I think it could be a another interesting application thanks Pablo I think that's a good place to kind of wrap this up you know pointing out we've got a lot of good work in this space but also still lots of opportunity support you know both the patrollers and the editors and we'll wrap this up thanks Isaac and thank you for the presentations this is a great showcase I'd like to close out with a few words of thank yous to I told King Diego and Pablo for presenting today and this showcase was also made possible thanks to the coordination on the team with my colleagues Isaac and Pablo so thank you both and Isaac thank you for taking the questions Emeril thank you for the AV support and Janet for the coordination support thank you to everyone who watched listen and ask questions again there will be no showcase in August because of Wikimania so the next showcase will be in September our theme will be rules on Wikipedia and this is in response to the attention to this topic received in the wiki workshop so we're looking forward to seeing you all there thank you everyone


Vinnie box, but I gotta take that box. How about I just be a good bird? I said no box, okay? Come on I'll give you your own special box if we go back. Nope You're so naughty.


Who do you think is going to win the college football playoffs? Oh, the whole thing, right now? I watched the order play. The order game is looking pretty good. I have a rematch, Alabama, Georgia. Who's winning? Maybe Alabama gets revenge from last year. Georgia. I agree. I think so too. Okay, that's what's up. I think so too. I'm hoping my Florida State Seminoles are back this year, so they can sneak in there, if they can win some decent games this year. It's really up in the air. These kids play so hard and they have so much fun doing what they do. I'm just happy that they finally are going to expand the teams. The best 12 will go and play. That's going to be pretty lit.


I'm divorced. I was divorced when my son was five. We split, separated. My daughter six, he was five. And very contentious, it's always, and it's still contentious to this day. Her grandparents, I saw my children's grandparents on the mother side, we were at my daughter's tennis match. By this time my son's 10, 11. He's growing up to be that man that needs to start to be a young man. Well, the grandparent in an attempt to try to keep the son away from me during these things, you know, they do these microaggressions, death by a thousand cuts. They won't let the kid just roam and be free.  The grandmother takes my son and puts the son on her lap. That's her attempt to destroy him as a man. I don't know.  You might be thinking as a parent, well, that ain't that bad. Go back to when you were 10. You were probably fifth grade. Did you want to be sitting on your grandmother's lap in public? That's a emasculation tactic that people use, but it wasn't towards to emasculate him. It was to punish me, but she was willing to emasculate him by doing that. So I said, no, you're not going to be sitting on anybody's lap from this point forward. And I got them from that situation. They called the police on me because that was inciting. That was some sort of violation of custody order and so forth and so on. This is how women emotionally punish young men to get back at the father.  I don't want to know. Oh, he left. That would be the story. He left. See, this is how men get out of the situations. You don't fight hard enough. We don't put up with all that microaggressions. And then when they do a microaggression and I say, well, she had my son sitting on his grandmother's lap. What's wrong with that? We can't lead. We need you to lead, but not like that.  That's men. We've allowed this stuff. Okay. And so the mother will create this close bond with the child. But the child won't be allowed to create this bond with the father. This is called in Sigmund Freud, whether you like him or not, called this a part of the Oedipus complex. And we're going to go through a couple of stages. He has five stages. We're not going to go through them specifically, but we're going to highlight a portion of the Oedipus complex that you might need to be aware of. I'm going to get there in a second. Okay. So according to Freud, a child must overcome conflict at each stage of his sexual development, because that's what we're doing. We're maturing sexually. Okay. And this is for him to be able to develop healthy sexual desires and behaviors. Now, this is actually true. Most of his context was related around sexuality. People aren't uncomfortable about that because we live in a Puritan society where we actually believe that sex doesn't happen. The stork delivers babies. We try to avoid the conversation of sexuality, which actually stunts our growth, but I get why people try to protect people. But we go too far with it. And Sigmund Freud was talking about these sexual relationships and desires as either healthy or unhealthy. And I can pretty much attest that people have a lot of unhealthy sexual behaviors. And that comes from lack of maturity. Pimp culture.  Because much of the stuff that they display and prioritize is around sex. That sex makes them the man. Okay. Many of these pimps wear flashy clothes like women. They have hairstyles like women.  And when I tell people this, ooh, people get mad at me because we uphold this individual in our society as having some sort of success. But he's actually in addition of the he's an extension of the mother, son husband, surrounded by aunts that like to get perms. And he was a young boy watching this and them being happy about perming their hair. And so his dumbass gets in there and he perms his hair to fit in or he braids it in cornrows, which in the 90s was a male's hairstyle. In the 80s, it was a girl's hairstyle. But because men didn't have fathers around, their little sisters, the aunts, mama and grandmama braided the son's hair.  In that example, that is an extension of an absent father. Okay. Now, in the edifice complex, it says right here when the edifice complex is not successfully resolved during the phallic stage, an unhealthy fixation can develop and remain says right here, this leads to boys becoming fixated on their mothers and then girls being fixated on the father says right here. And this causes them to choose romantic partners that resemble the opposite sex parent as adults.  That's kind of weird. You know what I mean? Because I never would look at my mother as a romantic partner. Now maybe you might be talking about everything else, but if I choose a partner sexually, I would have to actually go there with that partner. And if she's like my mom, we also do this with daughters as well. Okay. She'll end up choosing a man just like her father, right? We kind of do that as well. And that's what Sigmund Freud was talking about. Now here's where the problem leads. Here's where the problem leads. So there's Newport dad trying to get in again. Dad trying to come back. He's trying to come back all the time. All right. So anyway, he comes back when I get drafted in the NBA. And then he's like son million dollar son. Okay. But here we go. Now it says right here when it when the boy is between the age of three and six, this is development. He becomes unconsciously, sexually attracted to his mother. I know that's uncomfortable, but he was just breastfeeding on the woman. Okay. So we have to understand that this is actually happens and he may become hostile towards the father and looks at the father as a rival. I'll give you an example because it sounds weird.  What would the kid typically do?  The kid would probably look that's my mom. She's mine and he might break that up. No, you can't be kissing on that. No, no, no, no. What is this? I'll give you another example. If you ask a young kid in this age, who do they want to marry? Who are they gonna say? If it's the son, he'll say, I want to marry mommy. If it's a daughter, I want to marry daddy. Why? That's their world. That's all they can see when they go out in public. Everybody other than mom and dad is stranger danger. They're almost invisible.  So they become fixated with mom and dad. They don't know anything about sexual dynamics between two people. They know nothing. They just want to marry mom. Your son just want to marry his mama. If you're in a custody battle and the child's between that age and you move out to another place and the son has to be dropped off for your custody exchange, that kid might cry, I miss my mama. And then the mom's like, see, he don't even want to go over there. This is because they have this fixation with the mom and vice versa, the daughter with the dad. And sometimes this is actually used against the dad. He doesn't like to be around a dad. He cries when he's with the dad. See, these are something that happened. Here's the thing right here. Indie and jealousy are aimed at the father. This is in reference to the son. He may become jealous of the father and envious of the father because he has a close attachment with the mom. He may want to break that up. And the father is the object of the mother's affection and attention. He should be. We're kind of getting away from that as well. But that's a whole another conversation. So if the father has a good sexual or healthy relationship with the mother, the son might, hey, what's going on here? Okay, that's my woman. I came from her, right? I get fed through her. And it's right here. These feelings for the mother and rivalry towards the father, it leads to fantasizing him getting rid of the father and then taking the place of the father. You don't like your daddy, do you? He mean to you, huh? He spanks you. He disciplines you.  Oh, he don't like the father. And then this can happen. Okay, some people have this hyper experience. Some people have it micro. But this is actually what happens in our culture. Now, this is the important part.


It's like, or it's like, right on, with my foot.  It's picking up now. Oh, there we go. Yeah. Testing. Wait. You hear us? That looks like it's working now. Yeah. Yeah. What? That was literally just, I had properties on that and it worked. And it started working. Why are all the problems in this place just like one simple button? Every single time. That's so weird. It's like, hey, I have a problem. Oh, I just turned it off back on. Geez. Rich sentence doesn't sound good though.  Let's see. Let's see. Maybe it is. It is. One sec. Any better? Sound better. It should be under cable A. Cable A though. But super loud. You just turned off the game. Some tons of game. A little bit better. How does that sound? Hard to gauge. Yeah. I'm going to get right here too. It's a little weird. It looks like close to mic and I have it sound good. Muffle a little bit still. Muffle a little bit. That change. I didn't really change anything. No. Crop something. Expression. Define. I actually have no idea why it would be like that. It's weird. If you're 5, that's going to be like a QI suppressor. A suppressor. But we aren't. I don't think we are.  There is a noise suppression filter. That's all I see. That's the only filter that's on. Yeah. That's the one that's on. Expression level is a minus 42. It sounds terrible. Let me go see how it sounds and then I can probably figure it out. Sounds good. We got Will working on it. Thank you, Will. Interesting. That's so weird. Don't think it's on the main mic. Like it's using a camera mic. A camera mic says it's muted, Rich. Yeah. Weird. It does sound like muffles. Muffles? Like almost like it is using a different mic. But it's not. I'm looking at voice mod right now. Yeah. It's muted on the area. The mic aux 3 is the one that the camera picks up. Let's double check. Because we might have changed settings here and now we have some audio. That makes no sense. This is the wrong microphone. Is it? I'm figuring it out. One second, everybody. I'm trying to figure it out. This is acting very strange. We're getting there. That says no audio now. Yep. It's going to have no audio. That's true. Yeah. I'm just going to keep talking. Switches out exactly. Something's not right. That's so weird. Yeah. Thanks. Audio is on now. Yeah. Rich says. Yep. Okay. Switching it over. It says it sounds the same as before. Default audio is the voice mod. Default audio voice mod. So switching it over to output. Because that's the audio. Yeah. It's the same thing. There should be zero turnout voice mod. But then, let's see how that works. Put the output on the main mic. Testing. Output to the main mic. How does that sound? Testing. Probably not the one. Yeah. There's no audio. Yeah, we're aware of it. Yeah, I've been muting and unmuting the mics. Yeah. Trying to figure out what's going on. Like right now, I understand that it's not capturing properly. But if I change the output, it's going to go mute like right. Default, it works. But it's picking up the default is something else. Yeah. How does this sound now? How does this test thing test thing? How does it sound? How does it sound? Good. Good. Not good. Maybe a little bit good. Not right at all. Yeah, I don't know what's happening. Like I might now just set it back to the input device on the computer is this microphone. Just this microphone. And then, but this is under the same input as it was the voice mod. All right. We have to restart. The entire computer? I would say the entire computer. All right. Sounds good, guys. Give us one second. We're going to restart.


You may be required to use an evidence-based research article. The best way to determine whether an article is evidence-based is to be familiar with the different sections in an evidence-based article. Title, Authors with Affiliation, Abstract, Introduction with a Literature Review. This is where the authors say what their question is. Method or methodology, which is how they will ask their question. Results, which is the raw data. Discussion, which is where they analyze the data and say what they think it means. Sometimes there are conclusions or implications for future research. References that include all the articles they read for their literature review and maybe some seminal or super important works. I always scan the article for these headings to make sure I know what I'm reading.


Hello, and welcome back to the craft box. My name is Brenton. Today, we're in episode 28 of the Pirate 101 Privateer Walkthrough. Last time, if you haven't checked it out already, we made our way over into Port Regal. We got a new ship, and today we're heading into the SILAS. Aren't we just so spoiled? We come here and we get sent to the SILAS straight away. It's amazing, isn't it? I know. But that's what we gotta do. So what are we doing, Dad, today? I don't actually know what's in store for us. I can't remember. We'll find out. Again, this walkthrough has no dialogue. So if you want to hear dialogue, go check out the Musketeer walkthrough. We're basically just skipping all the dialogue, hoping to go quicker than, you know, just trying to get quick. Be quick, and good. Basically what we're trying to do. So yeah, we're doing that and things and stuff happen, and it is lit every single time we record, bro. Okay. So yeah, we, this shouldn't be too hard of a battle. We will hopefully wreck. Overwatch. Ooh, blocked. Unlucky. Ooh, dodge. Unlucky. Trying to hit, but nope. Gotta get dodged. Get dodged, son. Overwatch, yes. Okay. And overwatch. Fantastic. I love overwatch so much. Like, it's such a good epic. Such a good epic. Mega nice. That's a lot. That's good. We can also bomb again. Like that. We'll do that. And actually, can we go like, we can go like that. I'm kind of keen for that one. We'll get him out already. That would be good. Epic. Nice. Fantastic. Okay. He'll be dead. But he'll be gone. He'll be so gone. Like, so gone. If he isn't, I'd be surprised. Okay. Those three are half health, which is great. Fantastic. And you nearly did. Get blocked. Fantastic. Okay. What are you doing? You're moving up to me? Oh, dodge. Nice. Cheeky. Overwatch. Okay. They're nearly all half health, which is great. I am very keen for that to be a thing. Okay. Now they're just ganging up on people. It's kind of rude. Kind of rude, but we can deal with that. We're not too bad. Not too hard. We will probably go for a mighty charge on you. We will go you. You can go you. Actually, you can go on him. Try and get you out. Yep. Boop. Okay. And lance and blow. Fantastic. Okay. That's slightly awkward. Let's find it. Don't do too much. He's nearly dead. That's good. They had scored. Okay. He's low. First fire note. That's unfortunate. And he's also low. I'm, this is so easy. Cheap shot. Nice. Oh, he blocked that. Unlucky. Why you got a black, boy? Oh, the crits are real. Geez. Hello, critical. Need that heal. That heal would be good. Okay. Um, do, do, do, do, do, do, do, do. You.  See what that does. See how that goes. First fire. Nice. Oh, double tap. Yay. Fantastic. Yay. Super nice. Okay. Two down. And you were like nearly dead as well. So that's always good. Oh, you didn't move far enough. That's a shame. Uh, he did. Cool. One left. Fantastic. This won't be too bad. We will just do that, I suppose. Done. Fantastic. Dodge. Nice. Bap, bap, bap. Okay. And double tap. Nice. And Bonnie-Ann, can you finish it off? Surely you can. Yay. We did it. We did it. We did it. Okay. That was impressive. Here we go. Um, hello. Who are you again? You're Rex. We've been robbed. Seen as how they might not be. I wonder if you might tell the bosses for me. Let's go and do that then. Um, let's not get stuck. Getting stuck is a very bad idea. And that wasn't actually too bad. That was really easy, surprisingly. Um, I thought there would be so much more to do in the sewers, but apparently not. It's weird. But I know that we will definitely be back to some form of sewer really soon. So, hmm, oh well. It's fine. We will go talk to Gus or Sullivan, one of the two.  Stolen? Oh. We need your help, good captain. Deliver this message to Captain Corkeron informing him of the delay. Okay. Let's go do this. We're basically just running messages at this point. Um, yes, this is, just checking that it is actually the main quest, but it is. We just have to go and run messages. Uh, where is he? Oh, hello, Blind Mew. Hi. Hey, one gold. Oh my gosh. What a ripoff. Oh, here we go. We're going to be getting a new companion, I think, soon. Well, sometime soon. Oh, he's in the tavern. Hello. Another tavern. Yay. Got a love going to all these different taverns. Um, hello. Ugh, Bill Bobstay. I'll be back when I'm at Bill Bobstay in Scrimshaw. Okay, I am back. We are just about to enter the hole in the wall tavern. Oh, we don't even need to. Oh, wait, no, he's for a side quest, probably. We'll talk to him anyway. Um, yeah, it's okay. So we'll talk to Bill Bobstay, who I think had an appearance in Ratbid's promotion quest, so we already, like, talked to him a bit. Hello. Nick Deadeye, keep your voice down. Now, who's asking? Um, okay, hello. Bill, do you want to speak to me please? That would be great. Thanks. Best if go to the Pequod, the shipwreck under the jaws of the whale. That's where they meet. Ah, North East Plage. Aha, North East Plage, boy. Oh, I know it. Um, okay, that was awkward. Um, yes, let's go there and do that. That isn't too far away. It's just up this way in there. What's the bidding that Blind Mew is already here? He's already teleported over here. He's cheeky. He gets around, you know. Like, look, whoa, he's here as well. Geez. Whew. Oh my gosh. Um, what do you want? Hello. Um, we'll do that later. Thanks a lot for standing there. So rude. So rude. But let us hit on in here. And we will, then, see what we need to do next. I swear I have so many sidequests that I want to do. Like, there's way too many. I don't know when I'm going to do them all. Um, okay. It's Ratbid. What? Hey Ratbid. I don't like how they just re-use Ratbid's texture. That's kind of rude. Ratbid is a unique individual who should be respected, but apparently he's not unique anymore. Like, I don't mind if they use random war threats, but not Ratbid. That's just, like, what are you doing? Especially considering Ratbid is, like, so, like, pinnacle for, like, a whole arc of a story. Like, you just can't use them again. You can't. I'm sorry, you just can't.  You can't just reskin Ratbid and say, that's an enemy. Like, that's just rude. That's honestly not cool. Not cool. That's really not cool. I'm disappointed. Very disappointed. I didn't notice that, but if I did, I would have still been disappointed. So I'm just disappointed, you know? Um, we're going to defeat him quick, because I don't want to see him anymore. I don't want to see this fail of a character. That was really mean. I'm sorry. I'm sorry. Please don't hate to me.  Like, anything else I'm fine with. But Ratbid, seriously? You can't just reskin Ratbid. No. No. Nope. Sorry.  What are we going to do? We can probably bomb, but we can't. So that is unfortunate. We will do that. We'll do that. We'll do that. We'll do that. We'll go done. We'll go goodly. But okay. But fantastic. Burst first fire. Thank you. I swear Bonnie Anne isn't burst firing as much as, like, she used to. And like, other characters that I've had, she just refuses to burst fire. It's kind of weird. I mean, I need to give him a burst fire, I think. Why are you backing away? Weirdo. What a weirdo. What a weirdo. Weirdo. We'll do that. We'll do that. We'll do that. We'll do that. Okay. Hopefully that kills at least one of them would be nice. Burst fire plus. No, forget that's fine as well. Honestly. Boop, boop. Oh, gosh, you're nearly dead. That is not good. Dodge. Oh, gosh. Burst fire. Okay. Oh, gosh. Mega. Oh, gosh. Okay, we need to definitely team here. Geez. People are dying. These swashbucklers be OP. Swashbucklers in general are just OP. Um, we're going to heal you. You can team here. You can hit and done. Okay. Fantastic. Good. Okay. Fantastic. No, we didn't kill. That's unfortunate. Oh, yeah, there we go. He's gone. One left. And he shouldn't take too long to do. Done. We'll see how that looks. How much damage that does. And 26, okay. Wasn't too bad. There we go. Fantastic. Yay, we did it. We did it. We did it. We did it. We did it. We did it. Okay, I'll be back when I'm at Iago or found Iago. I am back. We have made it to hopefully where Iago is because that would be good. Very fast. Hello. Yes. Fantastic. So Iago is kind of OP. Doesn't have too many. He'll be fine. We'll be fine. We'll be fine. We'll do that. Sure. I'm just going to move up there and there and you can move up to there. Done. Fantastic. Good. Okay. Like that. And then what are they going to do? Okay, those fires don't really do anything. That's unlucky. Oh well. That did a lot of Gs. Okay. We might go for a cheeky wee critical buff. We'll do that. Sure. We'll do that. We'll do that. We'll do that. Just focus on him for a bit and we'll probably do a accuracy buff next. Oh nice. Super. Fantastic. That was good. Look at he's so dead. He's so dead. Oh he's dead. He's so dead. Yes. Okay. Fantastic. I'm impressed with that one. Woohoo. It does a lot of damage though. So that's just that worry. Nice. Okay. He's half health which is good. We will go for one of you.  That would be rude if they still did but I don't think they will. And double tap. Fantastic. And loop. And loop. Oh okay Bonnie is dying slowly. That's not good. Okay. We will probably heal Bonnie because she is so good. Dee Dee did. Yeah she is. We'll heal her. We'll do that. That. Done. Fantastic. Okay. And double tap into that. That's good. We'll get a lot of damage. Fantastic. And just two left. They're just going for melee apparently. That's also fine. Okay that did a lot.  Oh so close to dead. Yes. We will yeah we will team heal.  Fantastic. He's not dead this round but next round he will be. Okay. Do we have... Yes we do. Mighty charge. And he's dead. He's so dead. Fantastic. Okay. Look at that. Amazing. Doot doot doot doot doot doot doot. Yay. Oh where's the other one? Over here. Fantastic. Yay. Here's Mr. Sullivan's missing cargo. Let's get it back to him. I find that I'm tired of working for a filthy rat. You'll seem a far better captain. There we go. There we go. We've got Yago which is great. Um so yeah I'll be back when I'm at Sullivan. Okay I am back. We just got... Pirate you pulled us out of a hard spot. Yeah take this as a token of our assistance. Good thing I found you. Something urgent we need to talk about. Come save me and be careful. You must be working with a pirate. His ship is more so strong. Same thing.  Um okay let us mark out here and then we will go and see the Commodore. Come on. Thank you. And see what he wants to teach us. And then we will probably go and talk to a cat bed because cat bed is great. Love cat bed honestly. Such a champ. What a champ. Christmas presents yes please. Banana and potatoes okay then. Yikes. Let us head on in please. Let us in. Thank you. Officer training.  What's that? Oh yes of course. Yes we need some training. Absolutely you're just in... Pirate. That is cool. Oh and 25 accuracy. That's even better. That's lit. Um and a Spirit Decorps 25 accuracy in dodge. Fantastic. Elusive 2 at 32. That's fun. And then tough and then reinforced. Oh my gosh. We're gonna be good. We're gonna be good finally. What's that one? 50% incoming. That's cool. Um okay yeah I'll be back when I'm at cat bed. Okay I am back. We're at cat bed. Um yeah we're not listening to thingy. Oh nice wee profit show. Oh another wee profit show. Another wee profit show. Defeat 10. Excuse me. Who you? Who are you? Who do you think you are? Um so yeah I'm gonna end off the episode here. Next time we will be back after I've defeated 10 of these ships. So yeah thank you for watching. Remember to leave a like and subscribe. And if you haven't subscribed press the bell so you know exactly when I post another video. And remember craft outside the box.


In 1982, at the United Nations Convention, there was an Exclusive Economic Zone Act passed. This meant that each country owned mineral and fishing rights going at least 200 miles from the shoreline. So basically at the edge of each country's border facing the sea, the country had the right to go 200 miles out for resources. However, there was a catch. If the country owned islands that extended out past the 200 mile mark, well then that line was changed to the land mass of the island owned by the country. So if you owned islands, let's say, in the Caribbean, if you're the United States, then your land mass for oil rigs go out past the islands, not past the coast of the United States. Now, if you've been on this channel for a while, you know that I probably like you and not a fan of the UN. However, this 1982 act is what makes this story today so interesting. But before we go any further, you know what to do. Please hit that subscribe button and give us a like. Also, a very, very, very special thank you to all of our patrons and our producers on this channel. You guys are the true rock stars, and without you, this channel would not be possible. If you would like to join our Patreon community, there is a link down in the description box below. Welcome to Esoteric Atlanta. My name is Bryce, and today on Mystery Monday, we are gonna be talking about the missing island of Bermeja.  You probably learned the rhyme in school. That goes in 1492, Columbus sailed the ocean blue. History teaches us that Christopher Columbus was funded by the Spanish Empire under the rule of Isabel and Ferdinand. History also claims that Christopher Columbus was one of the first big explorers on this expedition to discover the new world. However, we do know the Vikings were here before. And if you've also been on this channel for a long time, you know that my personal belief is that the European powers that be already knew that this continent existed and were using it for a very long time. However, the 1492 story is the narrative that they teach children today. Nonetheless, Spain was a huge power that was then involved for the race for the new world. Alonso de Santa Cruz was a Spanish cartographer, historian, and teacher. He was born in Seville in 1505, dying in Madrid in 1567. Alonso de Santa Cruz worked on the Page-Drawn Reel. This was, and I quote, the official and secret Spanish master map used as a template for the maps present on all Spanish ships during the 16th century. It was made by the Casa de Contracción, which translates to the House of Trade. You see, when I say the race for the new world, that was what it was. Once this new world was found, all these European empires were racing to claim territory for their country. France, England, and Spain would all duke it out over the years to decide who controlled the resources found in this new continent. As you know, eventually England would take a stronghold until the Americans kicked England out during the American Revolution and took over our own country. The Mexicans would go on to also do the same in claiming their own independence. But during this race for the new world in the 16th century, many, many empires had secret maps. When the Spanish explorers would get back to Spain, they would have to go to the Casa de Contracción to turn over all their findings of new land. These explorers would have to swear under oath that the land they found was really there. And in 1539, Alonso de Santa Cruz added Bermeja to the list of islands that were located off of the coast of Mexico. In fact, this group of islands was right off of the Yucatán Peninsula. Now the Yucatán Peninsula is a peninsula that juts off of Mexico that divides the Gulf of Mexico with the Caribbean Sea. As you know, there are many islands in the Caribbean Sea and today in modern times, many of us flock to these islands for vacation. So what was so special about Bermeja? Well, first of all, Bermeja got its name because of its reddish tone that these sailors and the explorers claimed the island possessed. And this island existed on maps from the 16th century all the way to the 20th century. In fact, the last publication we have with this island included was in 1946. This was in a book called Los Islas Mexicanas which again included a list of Mexico's islands. But in 1982, with the UN's passing of the Exclusive Economic Zone, Bermeja all of a sudden became super important Mexico because you see the position of this island being at the northernmost point meant that Mexico technically had rights to all of the oil rigs out in the Gulf of Mexico and the Caribbean Sea. More oil meant more money for Mexico. And in the 1990s, a Mexican senator named Jose Angel Conchello insisted that Mexico needed to go and claim that island so they could then claim the resources from the oil rigs. There was one problem though. The island was missing. Without Bermeja, Mexico could not extend its 200 mile radius and the United States kept control over the oil rigs. So where did this island that had been on maps for four centuries go? Jose Angel Conchello, well he believed that the three-lettered alphabet owned by the United States blew up the island. Now in the 90s, that would have been a crazy conspiracy but nowadays this might be a leading contender as to where the island actually went. As much as I agree with this Mexican senator that the three-lettered agency isn't the greatest thing in the whole world and is actually quite problematic, I don't think that's what happened to the island. I believe there's a little bit more of a logical explanation behind the island's vanishing. Because you see, this isn't the first time the island has gone missing. The island's existence was first challenged in 1775 by a Spanish expedition. They had gone to the coordinates and found that there was nothing there but sea. This now brings us all the way back to the 16th century, to the Casa de Contracción and the Pedro Real. Again, as I said, there was a race for territory in the New World and cartographers, especially good cartographers were hard to come by. As I said in the beginning, this Spanish map, this template, was kept in secrecy. These empires did not want each other seeing what was on each other's maps and the only way for these empires to know if there were spies within their privy councils of explorers was to occasionally draw fake land. Then they would know if other maps showed up with these fake islands and fake lands, then they had a problem.  Many people believe that this was the case with Bermeja. After all, there is no photography of the island in existence. And as I said, the island was on maps up until the 20th century. So there should be at least one photograph of this island somewhere. So if Bermeja was a made up landmass to see if there were spies within the Spanish expedition fleet, well, they got their answer because many, many maps then contained this island of Bermeja. And only those in the top echelon of this map making empire of Spain knew that that island did not actually exist. But they were able to then suss out who was sharing their top secret information. Since so many maps were copied with Bermeja on it, it would make sense then that in 1775, this information would have been passed down leading Spanish explorers to believe there was an island and being massively surprised when they got to the coordinates and no such island existed. However, this isn't the only theory of what happened to Bermeja. There are others that think the island's disappearance was just that of mother nature. That the ocean slowly ate the island away. Some people speculate that the island was made up of coral and that's why it disintegrated into the ocean. In 2010, a group of Mexican researchers wanted to get to the bottom of what happened to Bermeja Island. They wrote a book called Donde Está La Isla Bermeja. Basically, where is the island Bermeja? These scientists came up with two conclusions. Their first conclusion was that yes, the island never existed to begin with. It was just a sleight of hand in the 16th century by the original Spanish explorers. But their second conclusion was that the island does exist. But it's just not where it originally was supposed to be. That the coordinates are off. They explained that this would have been easy to do in the 16th century. Being on a boat using magnets, not totally sure where you were, meant that a lot of times these coordinates were the best guess possible. So there is an idea that there is a Bermeja floating out there somewhere in the sea, nowhere near the original spot it's supposed to be. Or is it something else altogether? Was there a timeline shift? Was there something more spiritual that happened that pushed Bermeja into another dimension? All right guys, let me know your opinions down in the comments sections below. Do you think the three letter agency blew it up so that the Mexicans would not be able to get to the oil rigs? Do you think that this was just a sleight of hand done by the original Spanish explorers in the 16th century? Meaning that this island never existed in the first place? Or do you think that there is an island somewhere out there called Bermeja? That isn't where it's supposed to be because the coordinates were wrong. Thank you again to Josh McKay for doing our opening song. If you would like to purchase the opening song, again there is a link down in the description box below. And thank you so much to Todd Roderick for helping me get this video out to you guys today. I hope that you're having a fantastic day and I will talk to you soon. Bye.


My most complimented clothing brand, and it's not even close. Hey, what's up guys? It's Josh here. Today I wanted to do an interesting little video on the brand of clothing that gets me the most compliments. Now compliments is usually something that we look for in the fragrance community. Nobody really talks about compliments with clothes. I actually get very, very, very little compliments with clothes. Maybe you do. I don't know. Most of the time I wear something at least a little bit edgy. I was just thinking, you know, maybe people rarely get compliments on their clothes, but this brand, now I found them on Instagram. I gotta say brands on Instagram are kind of hit and miss. You know, I gotta say I was absolutely blown away by this brand. I purchased from them three, four, maybe even five times. I have probably 10 to 20 designs and I've literally gotten more compliments with this clothing brand than I have gotten in my entire life combined. So that should say something that I'm just absolutely blown away and that is the Wazz Shop, W-A-Z Shop. Now they don't pay me anything for this video. I just had to come on here and say it because I'm blown away by their designs. Not only are their designs really, really unique, I think you can't buy them anywhere else, but they just have a unique kind of pop to them. I mean the amount of compliments like everywhere I go, people tripping over me like young kids, girls, old guys.  You know what I mean? They do take a while to ship. Sometimes I would say the average shipping is probably about 10 to 20 days. I know that's not very good, but you're going to get it and you'll have the shirt permanently. You have the stuff permanently. They do a lot of deals. I don't know if they're kind of changing that now, but they usually have all sorts of deals. So if you kind of look on there, you kind of cherry pick the best deals for the right designs. And I think they do charge for shipping, but there was times where I think I was getting some of these shirts for like less than 20 bucks. I think sometimes the lowest 15, the quality is top notch. The design is unique. I don't know if you guys know, but most clothing brands, they buy a generic shirt and they go print on it. This brand seems like every single thing they do is unique, which is very rare. One of the other problems that they have, I'm not going to lie, is that they're sold out almost all the time. I don't blame them because the stuff is so good. Remember, I had a kid come up to me in the jam. He looked like he was 18 when I had a tank top on. And he's like, Oh man, that's the sickest thing. Like, where'd you buy it? And then I showed him where to buy it and I checked later and it was already sold out. So so many of the good designs are sold out. But I'm letting you guys know now if you're into this kind of clothing, I mean, I haven't come across anything as good. And I've seen other brands on Instagram kind of advertised and they look pretty good, but nothing is as good as the Wild Shop. And again, they're not paying me anything. I'm just letting you guys know I've probably got at least 30 compliments wearing these. Anyways, guys, we're on the road to 50,000 subscribers and I couldn't do without any of you guys help. You guys are the best. Have a great day out here. Hopefully, have a great day at home. See you all in the next video. Peace.