All right, thank you everyone for joining today. I know it's a busy time across the university with exams and finals and all of that, so appreciate you joining us. We're really excited for this penultimate installment of our fall speaker series. We're lucky to be joined by Tali Stroud and Josh Tucker, who will be co-chair to the external academic team on the US 2020 Facebook and Instagram election study. Tali is also professor in the Department of Communication Studies and the School of Journalism and Media at UT Austin and the founding and current director of the Center for Media Engagement and a co-director at New Public. And Josh is a professor of politics, affiliate professor of Russian and Slavic studies, and affiliated professor of data science at NYU. He's the director of their Jordan Center for Advanced Study of Russia and the co-director of their Center for Social Media and Politics. So we're really lucky to have them here today discussing a pretty urgent topic from cutting-edge social science. So I hope you'll join us or join me in welcoming them. Thank you so much, and thank you for hosting us today. We're just so honored to be here and really appreciate the invitation and a chance to share a little bit of the research that we've been doing as part of the Facebook and Instagram US 2021 presidential election. Can people hear me okay in the back and online? Are we all good? Just a little bit higher. Louder, louder, okay. Better? Yes. Okay, great. Okay. Well, let's go ahead and get started by acknowledging that Josh and I are two representatives of a huge team of people. Research like this takes a lot of different people. And so we just wanted to, at the outset, acknowledge all of the collaborators and contributors to this work, both at a variety of different academic institutions and within the core meta team. What we'd like to do today is first talk about the collaboration with meta, the decisions that we made there and how we came to do this research. Then we'll tell you a little bit about the research design. We're gonna focus on two parts of the results here. There are so many results across all of these that we definitely won't do justice to every part of these, but we're gonna focus a bit today on the on-platform experiences on Facebook and Instagram, and then we'll hone in a bit on polarization. And finally, we'll talk a bit about what we can and cannot conclude from this sort of research. So to begin, let's talk a little bit about collaborating with meta. It's a very interesting position to be in where the subject of study is also your collaborator. And so we wanted to really think through how we were going to engage in this research and really be thoughtful about that from the outset. A bit of background to begin with, the partnership with meta began in early 2020. There was a core team of 17 researchers that collaborated to design the overall project. And we added additional academic team members to fill specific needs on the various papers. We worked with NORC from the University of Chicago, and they partnered for gathering the data on the study. And the scoop of the study, the idea of the study from the outset was that this would be examining the role and impact of Facebook and Instagram in the context of the US 2020 elections. If you're curious more about the process of how this came to be, we wrote out an FAQ. It's publicly posted, you can search for it there on Medium. And we just tried to really give the details about how this collaboration came to be and how we thought about it from the outset. So encourage you to look there if you'd like more information. So when we approached this, we thought to ourselves, we really need to think about the integrity provisions that we put in place prior to initiating this sort of a collaboration and doing this sort of research. And the first thing that we did was we pre-registered the study designs. So this was including all of the research that we were planning to do on a date and time-staped stamp to website prior to initiating it. These pre-registerations are also available after the studies are published and of course as part of the peer review process. And this was a way to ensure that at the outset, we were specifying what we were going to do, that everyone was in agreement, that before we knew any results, this was the research that we were going to do and what we were going to present in the eventual papers. Before we began the study, we were very adamant that we wanted to ensure that there was going to be no pre-publication approval. We didn't just want this to be a situation where our finding was potentially problematic for the company and they said they would have some right of refusal, that was not the case here. So Meadow was able to check the research that we were doing first for its privacy obligations to make sure we weren't violating any of those obligations for their legal obligations. And then also in terms of feasibility, so that we came to them and said, let's do this billion dollar project. They of course have the ability to say, oh, I'm sorry, that's not feasible at the time. So those were some of the provisions that we put in place in terms of review. The way this was structured was that we had a variety of different pre-registrations that were put in place and each one we specified academic lead authors and these lead authors had control rights. So the event of any disagreement about what would be included in the paper, whether it came from other academics on the team or whether it came from the medical collaborators, those academic core authors retained the final control rights and they had the final decision-making power to say this is what's included or excluded in the paper. We also put into place a project repertoire. So not only were we studying all of these people and what was happening, but we were also being studied because this was Mike Weijker from University of Wisconsin and he was essentially an ethnographer throughout this project determining how things went and observing the process. He also has published some of the results of his work alongside the project results that we'll share with you today. We also have the data that are publicly available. So once the projects have been published, the data are included on an archive at SOMAR and ICPSR that this is at the University of Michigan and this is for researchers who are interested in replicating or extending any of the findings. So if you find any of the data here of interest, really encourage you to check out and apply. You do have to have an IOV approval, no sorts of steps in place, but it's all there so encourage you to take a look at that if there's additional research that you would like to do to extend what we're sharing here today and in the papers. So the process by which the paper topics were chosen, we agreed at the outset that the overarching context here was Facebook and Instagram and the US 2020 election and then we specified four key areas of interest that would govern the research project and so the academic team identified political polarization as the first one, the second political participation including vote choice and turnout, the third was disinformation, misinformation, information, knowledge and perception and misperception so we kind of lumped all of those together and then the fourth and final one was attitudes and beliefs about democratic norms this included confidence in institutions as well. So with that broad framework we then said about how are we going to work as part of this collaboration in order to identify the research projects that we're going to conduct and so we chose particular topics of study and then assembled research teams around these topics or around these ideas. We had some groups that were focused on experimental sorts of studies and I'll tell you a little bit more about those in just a moment. We had some that were observational studies so looking at what was actually happening on the platform and then we had some that were a combination of both experimental and observational and you'll get a flavor for all three of these as we talk through some of the results that we've been sharing so far. The first four papers were published this summer in science and in nature. Really encourage you to take a look at the full papers as I mentioned at the outset we're only going to be able to kind of brush the surface of the findings here but they're robust supplemental appendices with lots of details there that I encourage you to read. Okay so what was the research design? Today what we're going to talk about is these first four studies and first we have some observational studies that we'll share with you today. The first is some analysis looking at ideological segregation on Facebook and we look at ideological segregation in two ways. The first is looking at the measure of segregation or how ideologically segregated the audience is with a maximum score here meaning that conservatives are only looking at conservatives news and liberals are only looking at liberal news to the other end of the extreme which would be that liberals and conservatives are both looking at the same sorts of outlets and URLs and then favorability which is looking at the political composition of who is looking at political news URLs on the platform. Zero being completely balanced negative being liberal and positive being conservative. And in this paper what we were able to look at is all political news URLs that circulated on the Facebook platform and were shared by 100 or more shared 100 or more times on the platform. So we'll tell you more about the results of that a little bit here. We also have an observational study where we were looking at like-minded exposure and here we're looking at how often people are encountering like-minded sources and I really want to emphasize the word sources here because we're not just looking at content so we're not looking at content that may express the same ideological view that one holds. We're looking at friends, pages and groups that share the ideological affiliation of the individual user and so this doesn't only include political content it includes any content that was posted by a like-minded user, a like-minded page or a like-minded group and the reason we did this was theoretical because it turns out that even non-political things can send very political cues. When you say I'm going in my hybrid car to shop at Whole Foods you may not be intending to indicate your partisanship but you may be sending that sort of a signal. So in this study what we're looking at is how often are people exposed to ideologically like-minded sources? The experimental treatments that we'll share today are threefold and this was a really exciting opportunity to test some of these proposals that people have made about how the platforms might be changed and their really theoretical ideas undergirding what exactly is it that could be done on these platforms and what effect would it have? So the first is altering the algorithm and so in this study if we replace the algorithmic feed the ranking done by the internal ranking algorithm in use of the time with a reverse chronological feed where people then just saw the most recent content that was posted first and could scroll down and see things in the order in which they were posted based on the time ordering. The second one was to think about virality and so first altering the algorithm second thinking about virality and here is where we blocked reshared content so when people's feed we altered the algorithm so content that had been reshared wasn't included was demoted severely unblocked in most cases in the feed and then the third and final one is where we were thinking about the idea of filter bubbles and echo chambers and so here what we did is we experimentally reduced content from politically like minded sources in people's feeds. So these are three alterations that we made to the content that people were seeing in their feeds and we were able to evaluate then what the consequences were. In order to recruit people to participate in the experiment we recruited people that were based in the US that were age 18 and above and monthly active users on the Facebook and Instagram platforms. We recruited people in both English and Spanish and participants gave their informed consent to participate in the experiment so they were notified that it's possible that enrolling a part of this that their feed would be altered in some way although we did not specify precisely the way in which anyone's feed would be altered. They participated in the study for a period of three months and approximately we have around 6,000 people per treatment around 14,000 under control. The data that we collected first our primary data source was a panel survey that was conducted in between August and March surrounding the 2020 election and this was the line partnership with NRC. In addition we appended all of the survey data with a log data that individuals consented to include from both their Facebook or from their Facebook and their or their Instagram experience. And so this included things like how much time they were spending on the platform or what they were engaging with. And then we had aggregated Facebook and Instagram log data. This was for the observational studies so that we could understand what was happening across all US users on the platform who are age 18 and above. We also have data on passive browser and app usage tracking. So for a subset of the participants they agreed to allow us to track what they were doing on their mobile devices and on their browsers so that we could then evaluate what was happening outside of the meta platforms that we were examining and look at how that changed as a consequence of the changes that we made to the algorithm. We also have data on public records including voter files and FPC contributions. We won't be talking about that today. We also have Twitter usage data for people that agree to share their Twitter handle with us and then we scrape data to associate with that. Okay, getting into the results I'm going to start sharing a little bit with what we learned about how people's on platform experiences were altered and I'm going to focus predominantly here on the experiments that we did. So the first experiment that we did that I'm going to share some results for is when we changed what was happening with reshares. So this is when we're blocking reshares. This is when we're thinking about what is it if we change the nature of our reality on the platform. And when we do this you can just see by taking a quick glance through here it's not as though what they saw in the feed changed dramatically if you compare the yellow bars to the blue bars. So it's not like they all love political content before and no one's off political content afterward. But there were really important shifts that if you look at the relative difference in terms of percentage it was pretty noteworthy how blocking reshares changed what people experienced on the platform. So when reshares content is blocked on Facebook users see less political content. They also see less content from cross cutting and like minded sources. So they're less likely to see content that was shared by cross cutting sources and like minded sources. And in fact see slightly more content from either moderate or mixed sources. And what we mean by that is either people that are going to those sources like they're part of a certain group or they like a certain page that they don't have strong partisan identities or it means that the people that are going there are both from the left and the right resulting in an average being right around the middle. So that sort of content is elevated when we block reshares. When we block reshares people see less content from untrustworthy sources and they see more uncivil content. So you get kind of an interesting mix here. The solution of course is okay if we just block reshares maybe we dampen down virality and then this will be something that will help people on the platform. But if we just look at the content that they see we might say okay maybe it's a good thing that they saw less content from untrustworthy sources but they also saw less political content and they saw more uncivil content. So a takeaway from this and several of the other studies is the way in which these seemingly simple changes to the algorithm all to what people see it's difficult to come down on the side because this is normatively good or this is normatively bad in terms of the consequence. Okay for the experiment in which we change the algorithm from the standard algorithmic prioritization of content to the chronological feed we were able to look at both Facebook and Instagram and this is instructive as well. So when we switch to a chronological feed users see more political content. So that changes. We're only able to look at cross cutting like minded and moderate sources on Facebook. We don't have access to that data on Instagram because the platform doesn't have an ideological classifier in the same way for Instagram. But when we look at Facebook we see less content from cross cutting and like minded sources and more content from moderate and mixed sources. We see more content from untrustworthy sources and we see on Facebook less uncivil content and contact with slow words. However on Instagram it's a bit different where we see a slight increase in uncivil content and contact with slow words. So I think the takeaway from this again is that aid it's not a normatively clean story. So we might say, oh that's great that people see more political content. Maybe they're gonna be able to think more about politics but they also see more content from untrustworthy sources. So it's a mixed bag here and the solution of oh let's just remove the algorithm. May have some things that we find normatively troubling. And the other takeaway is that these algorithms differ. Facebook and Instagram are different in terms of what they're privileging. We also see that when we alter the algorithm it changes how people allocate their time. And the place we saw this most clearly is when we switch from the algorithmic prioritization of content that Facebook uses by default to this chronological feed. People spent a lot less time on Facebook and a lot less time on Instagram when we switched their feed to chronological as opposed to ranked using the standard ranking algorithm. And then there were substitution effects over the subset of people that allowed us to look at their browsing behavior. We could see did they change their habits in terms of accessing other sites. And what we see is that on mobile those people who were switched to the chronological feed on Facebook increased their use of Instagram. And on their browser they increased their use of YouTube and Reddit. When they switched to chronological feed for Instagram on mobile they increased their use of YouTube and TikTok. So there is people alter their behavior on other sites when we change the algorithm on Facebook or on Instagram. They also change how they engage with content. And so this that I'm gonna show you here is from the like-minded study. When like-minded content is denoted people engage less with content from like-minded sources and from this info repeat offenders. So the first one had to make sense, right? If you're seeing less like-minded content you would anticipate that there's less engagement with like-minded content. The interesting part though is that when people see content from like-minded sources so provided that they see it which they're much less likely to do when we denote like-minded content but when they do see it they're actually more likely to engage with it. So it's almost as though in this setting where people don't have as much like-minded content and when they see it they're especially likely to say oh I'm gonna click on that, I'm gonna like that. They have an elevated probability of doing so. Which is kind of an interesting phenomenon if we think that algorithms are learning from people's behavior seeing less like-minded content seems to motivate them to engage with it more. Okay that's a broad brush on some of the platform experience results and I'm going to turn it over to Josh now to share something about the polarization results. Great. Thanks Tanya and thanks everybody so much. I want to echo Tanya and inviting us to be here today. It's a great pleasure to be able to present these results at first time which has made such an impact over the years about what the rest of us know about the internet and society and so it's been, it's a really honored to be here today. So as Tanya said I'm gonna take over and talk about polarization. And the reason I'm gonna focus on polarization is that as Tanya said, there is, as Tanya said, there is, there's a ton in these papers. Like and these are the kind of papers that have like eight page papers and then 200 page supplemental appendices. So you should really dig in and if there are particular questions you're working on like take a look at them to see there may be data that's in there and there's many more papers that are coming from the project that are gonna have in particular some of them will have a lot more observational data that we hope will be helpful for everyone to research. But when I first got asked to give a talk about what we had learned so far across the four papers, I thought I would focus on polarization because polarization is something that cuts across all of these particular papers. And so this is, I think we can, we learn a lot from looking across, more than we look at any of the individual papers from looking across the different papers. So let me remind you again, I'm gonna show you things about polarization now that come from both the observational studies as well as the experimental studies. And so let's begin with the observational studies. And just to reiterate, remind you again what Talia told you that we did in these observational studies. We have two different ways we can look at polarization in the observational studies. We can look at ideological segregation and exposure to news. So that is for news sources on the plot. There's links to news on the plot comment. They're primarily seen by Republicans, primarily seen by Democrats, or they seem to be mixed, or primarily seen by liberals, conservatives are mixed. And then to exposure from politically-liked-minded sources. So here's what we find. This chart gives a very condensed thing about what we have learned over time, we learned over time, about how much people are exposed, how much segregation there is in exposure to news on Facebook. And the key benchmark here to know is that there have been a button, and Talia told you about this statistic here zero would mean it's seen equally by liberals and conservatives. One would mean it's seen entirely by liberals or entirely by conservatives. So this is a measure of how segregated the audience is. This isn't telling you what direction it's segregated, but just how segregated it is. The key benchmark to know here is that prior to this particular, prior to us running this study on Facebook, sorry, I'm just gonna set the timer so I know what I'm talking about. Prior to us running this particular study on Facebook, people had done some studies of this on browsers. So looking at what people left to their own devices, what kind of segregation did that lead to? And the benchmark for the browsers was about 0.10. So there was some segregation in people's news consumption habits. But when we look at the audiences on Facebook and Instagram, it's much higher than that. So here, the yellow line here is looking at the domain level, and the blue line here is looking at the individual story level. So the takeaway from this is that there are pretty high levels of segregation in regard to news exposure on Facebook. Oops, sorry. Moreover, we have the ability to look at this segregation at different levels. And you can think about this. So if you think about people's exposure to news on the Facebook platform, there's kind of a number of different ways that you could think about the level of segregation. The first is what we call the potential audience. So the potential audience for the news is everyone for whom that news could have appeared in their feed. And so where would that be? Well, so if I'm on Facebook, I can get things into my feed from what my friends post, what pages I follow post, and what groups I follow post. So we can think about this as kind of like a decision of the individual. The individual decides who they're gonna follow, and that creates a level of segregation. And that's our lowest level of segregation when we look at domains, and it's about the lowest level of segregation when we look at the URLs. Then we think of the next step, which is what am I actually exposed to? That's the algorithm. That is clearly algorithmic decision because you have all this potential content, but at the next step, it's well, what is the platform choose to show you? And we see that at the domain level that goes up, there's higher levels of segregation once the algorithm gets involved. Interestingly, at the URL level, there's actually no difference there. It's about basically the same. And then there's engagement. Well, this is clearly the human part of the behavior. It shows up on my screen, what am I gonna engage with? So what am I gonna click on? What am I gonna like? What am I gonna do something? And in both of these cases, there's another big jump when we look at engagement. So the highest levels of segregation that we find are in terms of what people are choosing to engage with when they get there. So in terms of who likes the story, who reacts to a story, that's even more segregated than who's shown a story. And that's across the URLs are all the domains. So what we see here is that when we think about this question of segregation, it is a complex interaction of algorithmic components from the platform, right? Which is definitely happening when we go from here to here. Human components, which is definitely happening when we go from here. And then there's this first step, which I described as, well, that's kind of a human component in terms of who you choose to make your networks with. But of course, we know these platforms also suggest who should be in your networks. So this first level is actually a component of both algorithms and human choice. So this segregation is to take what's segregation there, second, it comes from this kind of complex intersection of algorithmic effects and human choice. The next big thing we learned from this is that it's actually asymmetrical. So it turns out there's lots of segregation out there. And there's segregation on the left. There's plenty of stuff that's seen primarily by the left and there's stuff that's primarily seen by the right. But it's not symmetrical. It actually turns out, this is especially clear when you look at the URLs, that there is just a corner of the Facebook ecosystem where things are seen only, where news is seen entirely or almost entirely by people on the right. And there's something like that on the left, but it's much, much smaller. So you see the sort of when we get into like 95% of content only coming from the left is almost dropped off to a small part. On the right, this goes up much more tired, which has some overlaps with work that's been done at Berkman Klein in the past looking at the television ecosystem. All right, but we also have really interesting data from the like-minded sources paper at the object. Yeah. Just to clarify, how was the news content defined? How is news content defined in terms of being, whether or not the news content is on the left or the right? Okay, so this, oh, how was you? Also, like, is this news or not? What is news? Yeah, so we had a, there's a classifier for what was civic news. It was like a classically, you know, like training machine learning algorithm. This is all about audience determination. So everything I'm told, thanks for clarifying this. Everything I'm telling you here in these, in this study, in the Gonzalez de Bayon study, is all about audience determination. So this is, we have estimates of people's, people whose ideology we're able to estimate on the platform. We aggregate up to all the people who saw, interacted with, were engaged to these kinds of things. All right. So now the second place that we have observational data has to do with the like-minded sources paper. This is the nine-handed all paper. And in the like-minded sources paper, we're able to look at how much content in people's feeds comes from people who, from sources that are politically like-minded. So if I'm conservative, how much am I seeing from conservative sources? Cross-cutting, if I'm conservative, how much am I seeing from liberal sources? Or moderate, mixed, if I'm, you know, how much am I seeing from mixed sources, regardless of whether I'm conservative or liberal? All right. So that's the set up here. These are cumulative distribution functions, which are kind of interesting. I'm going to kind of give you on the slide the sort of takeaway from this, which is that, and here's the sort of main headline feature here, which is that for the me and Facebook user, right, like, so that's the person who's, you know, right here on this study on the 50th percentile, for our media and Facebook user, they're seeing a majority of their content from politically like-minded sources. So just over 50% of the content. That's in contrast to the media and Facebook user is seeing about 15% of the content that they see on Facebook from cross-cutting sources. But it's important to remember, so Talia was talking about earlier about how we made this decision to look at all content and breaking it down based on the political congruence with the source of the content.  So most of what you're seeing isn't political content or political news, but it's still coming from politically like-minded sources. So that gets into, if you read Jamie Saddle's book, Freight Fronemies, like this whole concept of like, how much of the, I'm driving my electric vehicle to the food market, you know, how many other subtle cues are we getting in there? What is interesting though, is that extreme echo chambers don't seem that prevalent. So only about a fifth of the sample is getting 75% or more from politically like-minded sources. So four fifths of people on Facebook are seeing at least a quarter of their content are coming from non-politically like-minded sources. But that being said, getting a lot of content from cross-cutting political sources is also pretty rare. So only about a third of people get at least 25% of their content from cross-cutting sources. And as I said, the median person only gets 15%. All right, so when we look at the observational evidence on polarization, Facebook definitely shows signs of echo chambers and political segregation. And I've been involved with a bunch of empirical papers that have pushed back on the overwhelming kind of echo chamber narrative that's out there that's shown that these social media platforms are actually a bit less of an echo chamber than people originally thought. I will say me personally working on this project based on this observational data, I updated my prior of it in terms of how ideologically segregated I think Facebook is. A lot of news is primarily seen by those on the left or the right, but not both. And this is especially so for news seen by the right. New segregation is higher on Facebook than in previous studies of web browsing. And it's much higher from groups and pages. I didn't mean to show those results those are in the paper here. Similarly, people are exposed to much more content from politically like-minded sources. And there's this complex relationship that seems to be driving that between algorithms and human choice. All right, now what about the experiments? Here's the big punch line, right? So we had these three experiments getting at these three big fundamental concepts. I don't have to tell you this at first, I'm fine. In the study of internet, morality, engagement maximizing algorithms and echo chambers, right? Like these are big things that we look at and think about having, we as scientists are really interested in the causal impact. We think these are fundamental features of social media and what's different about social media, the ability of information to go viral, people being able to self-select into echo chambers and these engagement-activizing algorithms. And we were able to really precisely manipulate features of the platform that are tied to each of those things. Holly went through all of these with reshares, replacing the algorithm with chronological feed and depressing the amount of content that people get from like-minded sources. And the findings were contrary to what we expected to find. There was no impact of any of these experiences on affective polarization. Those of you who are like political scientists use this term for how much you dislike the other party or ideological polarization, exactly how polarized you are on issues. In neither case do we find this. I'll just roll you through where we can find these things. But like this is the algorithm to reverse chronological feed. And remember, this was being proposed as a policy adaptation, right? To deal with polarization. No impact on Facebook or Instagram. These are really big experiments. We had 23,000 people in it. We were precisely powered to be able to detect nulls of less than 0.03 standard deviation. So this is much higher than any other story. So when we say these were no effects, these are really precisely powered to be able to reject the fact that there was an effect here. This is what we ended up finding with the reshares when we look at affective polarization and issue polarization. Again, no effect. This is what we found in the like-minded paper. Again, affective polarization or having extreme ideological views. No impact on it. Okay. So what can and can't we conclude from these studies? All right. So there's a ton of caveats I want to give here, right? Like we did these studies for only, we can say only three months. Now as a political scientist, like most of what we know, a lot of what we know about political behavior and the causal impacts of political behavior was based on getting University of Michigan undergraduates in a lab for, you know, an hour and a half. So three months is long or it's short, depending on your interest. We thought three months was a really long time. But of course, you know, maybe this is a big thing. If it's, maybe we find different effects if we're taking people off, you know, changing these things for two years. We only did it in one country and the United States has a weird political system compared to the rest of the world. So we don't know how this would play out in multi-party systems or single-party authoritarian regimes. This was during a period of heavy political exposure, right? And so, and there was a reason it was during a period of heavy political exposure. The entire purpose of this study was that we walked out of 2016 when the country kind of collectively lost its mind about the impact of social media on politics with so many unanswered questions. And part of our motivation for doing this study was we wanted to make sure that policymakers, scholars, the press, the community, the public would have more information about what actually happened on social media in the context of 2020 than we did in 2016. But this means that these results I'm showing you, this is a time when people are getting political information on TV. They're getting it on the radio. They're getting it from their neighbors. They're much more likely to be talking about politics in the hallway than they are other periods of time. So we can ask the question of, well, what if we did the same study not in the middle of a campaign season? Why did it have a difference? You know, when we think about these kinds of experiments, we have to think, you know, if I was in the treatment group, I wouldn't have seen reshared content. But Talia might be my friend in real life and she might still be seeing reshared content. And so, we didn't shut off reshared content for everybody, right? And there were ethical reasons why you don't change, you know, for 99% of the platform and legal 1% holdout, which is a different way you could have designed these kinds of studies, but maybe that would be different if we did this. Again, we privileged scientific inquiry in these studies. That was the point of this. And we were, this was an opportunity we had never before had as external scholars to look at particular aspects of the platform experience because we could change those particular aspects of the platform experience. So we did one at a time in each study because we wanted to get, we wanted to look at morality, we wanted to look at echo chambers. But maybe we would have seen something different if we changed all of them as opposed to only changing one of them. And then, if you guys can make this go away, my last point, it was only one platform, right? And people live on lots of platforms, right? So, and even the studies that we did on Facebook and Instagram, those were separate studies. We, again, to be precise about what we were studying, we didn't enroll people in both the Facebook and Instagram study. But it's possible if you change the way people were exposed to the content on Facebook and Instagram and TikTok and Twitter, you might see something different. All right, so what can we conclude? We can conclude that algorithmic changes affect platform experiences.  We can conclude that Facebook has a good deal of ideological segregation. As I said a moment ago, a lot of news consumed by liberals are conservatives and this is driven more by pages in group than users and people are getting much more content from ideologically like-minded sources on these platforms than they are from cross-counting sources. This, to me, all seems incontroversial. What we also can conclude is that we have not found simple answers for complex problems. We would love to have been able to come to you here today at Harvard Law School and say, look, get them to turn off reshares before every election and it will make people hate each other less. Like, that would have been great. It would have been a really good outcome.  But what we found is, and we think we went after here, like the big things that people are speculating, writing about in the literature, that people are speculating about in the press, that policy makers talk about, that people testify before Congress about, morality, intention-adjusting algorithms, you know, echo chambers, right? We went after the big ones and none of them seemed to give us confidence that if we just changed this for a couple months before an election, it would have a different impact. And I think for my case, I come from a background and I'm a political scientist who's been studying political behavior for my whole career, which started like over there. And I think some of this part to me is like there's a little bit of like revenge of political behaviorist here because political behaviors will tell you it's really hard to shift things in great attitudes like affective polarization, right? And there's been a lot of stuff that's come out of the internet research thing that says, man, this new thing in the internet is having this huge impact on things and we should just change it and things would be fixed, right? Like polarization is obviously coming from a lot of different angles in society. And again, it would have been great if we could come to you here with a solution and there's still projects ongoing and maybe some of the projects ongoing will allow us to do this. But it may just be a reflection of the fact that like there are big societal factors that have gotten us to this point in this country and I can speculate on what those are from my political science hat that have led to these high levels of polarization that we see in the United States, that we see in Europe, that we see in India, that we see in lots of places there are big global phenomenon collecting this and simply changing one aspect of people's experience on one platform is not gonna be enough to sort of deal with these kind of problems. All right, what can we not conclude? And this I wanna be really, really, but really with clear about to end on this. This is the question that everyone wants answered is is it social media's fault that we're living in a polarized world? That's a different question from if we change aspects of social media, can we make things better? We can't answer that question. Nothing in these studies should be interpreted to say if there hadn't been advances in computing power 15, 20 years ago that allowed these social media platforms to come about, if they hadn't, you know, if Mark Zuckerberg hadn't, you know, met somebody in the dorms over there and this that and the other thing, we still don't know the answer to those questions. Like if the world had done this differently, it's the wrong counterfactual. We can't rerun the last 15 years and any study we run now will be run against the background of a world that has had social media for 15 years. By the way, I think this is a very important point about AI as well. We start to think about the impact of AI in society and we start measuring these things. So consequently, we cannot conclude from these experience these experiments that social media has or has not had an impact on the growth and political polarization that we have seen globally and that we've seen in this country over the last 10 to 15 years and to claim on the basis of these experiments that that's the case is claiming too much from the basis of these experiments. And unfortunately though, the null results don't allow us to pinpoint a particular feature of social media as a potential candidate for exacerbating polarization independent of the larger information environment. So thank you so much for your time. We're super excited to answer any questions and discuss and I hope this has all been informative and thanks for all you guys do here at Berkeley Mines. So thanks so much. Awesome. I'm gonna selfishly offer the first question. One of the observations that the project rapporteur noted in his piece of science is that from his perspective the Meta's data architecture is so massive and complex and that as an outsider coming in it can be hard to know just the universe of observations and things you could get your hands on. So I guess it would be awesome if you could speak to that and have that shape to the questions you were trying to tackle the designs that you were coming up with. And then also taking a step back, looking at things like pad or other transparency proposals that are out there. Do you think those have enough teeth to ameliorate that sort of concern and coming out of the project what sort of provisions are really necessary to make sure that independent outside researchers are able to tackle the questions in an informed way? You wanna take the first question? Sure, have to. It absolutely is the case that being an outsider coming in to Meta, there is all sorts of things about the infrastructure of the data and what's stored and what isn't, what the retention policies are and how things are operationalized that you don't know. And so there most certainly was a learning curve and a lot of back and forth where we would say we're thinking about this and they would say, okay, well we have it in this way and okay, that's. So I think that that's an important thing to take into account is these are infrastructures that have built up over decade, right? And data infrastructures that have been developed and the way in which the code is stored and how the tables are populated and lots of bureaucracy in terms of what is, what is captured and what is not captured, what is retained and what isn't. So I think that that's a really valuable lesson actually for anyone that's thinking about doing this sort of platform research is how complex the data structures are. I mean, just imagine how many people are in there and how many things can be recorded in any given minute. And maybe I'll just say very briefly about the second part of your question and then pass to you, Josh, is I think that this makes clear that we need to have some infrastructure in place to make it possible for those people who are outside of companies to evaluate what's happening on them. But I think it also makes clear that you have to have some level of expertise on what's happening on the platform in order to do that because there are all sorts of parts to the infrastructure that you wouldn't know if you didn't have some of that background. Yeah, I just want to add on that second point. This is something that I feel passionate about and I've written about a lot. I think that, I mean, just pat it unless T, pat it hasn't been passed yet, right? I feel like, let's start with pat it, right? Like, you get that pass. But DSA is super exciting. The fact that these are kind of models, you know, we're really hopeful that maybe the platforms will respond to DSA by saying, well, we have to make this data available in Europe. We're going to build APIs that make it available more generally. It is absolutely clear that we live in a world right now where if you want to understand social media's impact on politics or on society or any of these things, that no matter where we go, we're kind of at the whim of the platform, right? Like, and Nate personally and I, in the conclusion of our book, Social Media and Democracy in the State of the Field, like, in our concluding chapter, we wrote like, look, you basically have four options as a researcher, right? Like, one is you work around the platforms and you don't collaborate with them and you try to get the data any way you can and that has pros and cons.  One is you collaborate with the platforms and you can. That has pros and definitely has cons, right?  And the fourth is that you just give up and say we're not going to try to go with these questions. And if we think the questions are so important that we don't do that, we're, you know, we're left with the other three. I got to say when we wrote the book, the one that looked like the long shot was like getting governments to get involved in regulation and making data available. The, I've been around that has really improved in the last few years. And, you know, and I know thanks to work that people are doing here at Garmin Klein and HLS, but like that's gotten a lot better, I think. But, but to your teeth question, right? Like, I think one of the things that have come out of this project is that it's possible. Like you may just agree with particular ways of ways that we did things. There might be, we have ways that if we were doing this over from scratch that we think we could improve this. But the big picture is it is possible to do a thing like this where you are able to run causal studies on super important issues, right? That can only be done by changing things about the platform experience itself to gather observational data to be able to see at the scale of the platform what is happening and to get this data out there. And, you know, my thought is, is that I hope is, is that this project will have two impacts, right? One is the direct impact. Everything that's in the papers we now know, we already know way more about 2020 than we did know about 2016. And the fact that this data is available for replication when people get in there, people are going to dive in and get more stuff as people look at the appendix. We have more papers coming from this project. That's the direct impact. The indirect impact is the model now exists. And so your question doesn't have enough teeth. I've heard people who are saying, who are, you know, super enthusiastic about DSA in Europe, responding to these studies when they came out and saying, wow, even with DSA, there's nothing in DSA to compel the platforms to allow people to run these types of experiments. And I think the model to think about is stress testing banks. Like in this country, we have no, I mean, I'm in Harvard Law School, so I'm there, people who know this only time more than I do. But we have setups where people are, you know, where it's said, this is so important. The bank is so important for society that they have to submit themselves to these kinds of tests on a regular basis. So we know what the impact of X and Y is on the banking system. Why are we not saying that, you know, about the information environment system? And so my hope is that the secondary impact of this study is that it's now a model. People can look at this, and if you're sitting in Indonesia and you want to know what the impact of Facebook is on Indonesian elections, maybe why are you sitting there and saying, hey, we should get to do this too? Like, why is this only being done in the US? So many viewers and listeners online trickling in with questions. I'm gonna try to summarize the question of Renny and Jonathan, which is somewhat connected and has to do with this issue of pre-hardened politics arriving at polarization. So Renny is asking, are you saying that rather than alcohol manipulation has negligible effect of impact, it's possible for people arrived in the political season with pre-hardened positions and therefore are less likely to be affected by manipulation or variables and those harder positions may have been facilitated and exerted by the social. And then Jonathan's asking the polarization, specific questions about the polarization study. One is in finding limited impact on polarization, how did you control for the amount of prior exposure to polarized content which might have caused someone to harden their beliefs already? And two, in saying only 20% of people got 75% plus of content from like-minded sources, did you look at people's propensity to vote for actual voting? Because only a subset of people vote even a small percentage of echo chamber, people could have been this proportional electoral effect. So let me try to answer those really quickly. So the first question, yeah, that's the answer is yes. Absolutely, thanks for the questions. That's what I was trying to say at the end. What we have learned is that changing these features of the platform experience for three months around the 2020 elections did not have a noticeable impact on polarization, bull stock. Could that be because people arrived at this point with very hard and polarized attitudes? Yeah, everything about me is a political behavior person thinks, yeah, polarization is pretty, these are pretty hard and attitudes most of the time. And they're hard to change anytime. Could that be because of the last 15 years of being exposed to social media? Sure, but I don't know, it could also be about the Great Recession and the aftermath of the Great Recession. It could be about political elites in this country and the type of rhetoric they're using. Like there's a lot of things that could be causing it. I don't, from this study, I don't know the answer to that. On the, how do we explore prior exposure to content? So there's two answers to that. One is, that's the beauty of experiments. When you randomly assign people to treatment and control, as long as your randomization process works, you should have that in both the treatment and control groups. And so the effect that you're trying to measure is independent of it. We also, however, pre-register a large number of heterogeneous treatment effects, which is what this is a class of questions for. And in these, that's way too much for it because there's heterogeneous treatment effects across all the papers and other things. It's way too much for us to get into in this kind of talk. I encourage people to see the appendices, to go look at the papers, if there are specific heterogeneous treatment effects you're interested in. But, the TLDR on this one is, It really quickly, because we did look at people's prior exposure to like-minded content, for example, as well as heterogeneous treatment effects and TLDR, and TLDR. And TLDR is that we didn't really find any heterogeneous treatment effects. Like we looked at tons of heterogeneous treatment effects and we found almost nothing. And so you can go back and look in and see these particular ones. In this particular case, I think we definitely didn't find anything because it would have been, it would have been something that was highlighted in the paper. But all that data is there. And if there are heterogeneous treatment effects that you didn't test for, you can go and you can go get the data yourself and test for these other types of heterogeneous treatment effects you want to test for, in the case. And just adding that we did look at voting in some of these and there were all several effects. So that's included. Do you guys just want to bunch your questions together? Sure. So thank you for being here. I know you talked about the system designer, just the platform itself. But what do you think, if anything, this study or these series of studies show and for content creators, in regards to trying to make engaging content? It seems to me that there are two Facebook related causal studies that precede you guys, one of which is Black Hat with Cambridge Analytica. And one of which is 15 years of AB studies for stickiness in algorithm designs, internal Facebook slash methods.  But did you get access to and compare the data on those two things? Did Facebook actually give you access to their AB investigations as part of the study? My question is at the very beginning, you said you were trying to study four things. You spent the rest of the time on polarization, so I don't know what the other three were. Did you get any non-null results on any of the other three things that you were studying? Take those, okay. Don't we just start with all three and then you can add on, okay. So thank you for those great questions. The first in terms of information for content creators, if you look inside of the studies when we remove reshares, for example, or when we switch to the chronological algorithm, you can see what sort of content is being algorithmically prioritized. So if you compare what sort of content is in the feed with chronological to what sort of content is in the feed with the engagement based ranking, if you compare those to each other, that's essentially telling you what the algorithm is prioritizing or deprioritizing.  So that would be one tidbit there. We did not have internal data on Cambridge Analytica or other AB studies that had been done in the past. Of course, you all know here that there's a trove of TINA here released, but it was a lower. So there's some information there, but we do not have access to that. And then the third question about other effects, in the reshares paper when we blocked reshared content, what we found is for those people in the sample, they were less, they had lower levels of news knowledge. So if you remember for reshares, there was less political content in people's feed when we blocked it. And for the sample itself, we see an impact in terms of news knowledge afterward. There's some indication there that it might be there for factual discernment as well, but once we adjust for the number of tests that we did, that effect is no longer significant. So there are hints of it there and both of those things point in the same direction. Then I think there's something going on there in terms of what people know. And the other two things you were saying, you got no results also. That is correct in the study. We didn't look at all outcome variables across all studies. So just with a caveat there, that there's more to be done there and it's not a perfect match, but across the board, mainly no results. Well, thank you very much. Congratulations for this study. My name is Dimitry Nasebova and I'm a professor in communication theory at the Complotencia University of Madrid. So first of all, I want to invite you, recommend you to repeat this study. As you said, maybe in multi-party countries or one-party countries. And also I would like to recommend you to do this in Spain because Spain is now a very interesting experimental field. We have a parliament with more than 10 parties. So my questions, how do you imagine would be the methodology in a multi-party, sorry, country, like the case of Spain, more than 10 parties in the parliament? I imagine that one of the difficulties has to do with ideology because it will be not the same. There will be cross-cutting topics between some of the parties. So the question is how do you imagine this study, thank you very much. Yeah, without going into a massive divergence in how we measure parties in multi-party systems which I would be happy to talk about for a long time. I would recommend that you take a look at the comparative study of electoral systems which is a cross-national collaborative election study project that piggybacks that puts common content and piggybacks on the backup of individual national election studies. I mean, I think things like affective polarization to the extent that we measure those with feeling parameters, you can measure that with lots of different parties. The question of how you measure ideological polarization in multi-party systems is much more complicated how you measure it at the systemic level. But you can look at things like whether people who are exposed to different types of feed treatments or something like that have more extreme positions on issues, that would be one way of doing it. It doesn't translate one-to-one to what that means to the overall system level of polarization, but it could get you an insight into that. One more listener viewer tuning online with the question, and this time it's Matt Mahon. There was one slide that you showed diversity of content on the left and the right, and the far side of the right soup was way more drastically than the left. Does this mean that the extreme right opinions or content are much further from zero than far left opinions and content? No. I was going to say, no, no, no. What that, just to be really clear about that, and I think the important thing to wrap your head around in the Gonzalez and Bion paper is that the unit of analysis is the article, the news article. So the unit of analysis is not individuals, individual people, and what we're looking at is the audience composition. So the maximum you can get is 100% audience composition. The composition is all liberals and no conservatives are all conservatives at no level. So what that figure showed, which is essentially a distribution of how that's distributed across the entire space going from 100% liberal and zero conservative to 100% conservative and zero liberal, it's spread out over the space. So there's plenty of content that is on both sides, but that peak at the right side, which is not mirrored on the left side means there is this bunch of content. And again, we're saying nothing about what's in that content. This is just the audience composition.  And then that's a, there's a chunk of things that kind of fall into that area. We don't see that a lot. That curve could look like this, right? It kind of looks instead, it kind of looks like this, right? And so you get across the distribution. So there's some content that's seen, there's some news content that's seen by both liberals and conservatives, but it could also look like this, right? If it was really that most news was being seen, who was seeing news with Lord Fogg on all the partisanship, then you would expect it to all be bunched in the middle. And that's clearly how that we found it. Just wanted to follow up on the content creator question. And for transparency, I used to be on the city of Cuba Facebook, so we work with some folks who know very well. One, I actually do think there are some opportunities on the production side to test some things out, especially on the reshares. So for example, what if there was an interstitial that popped up when you were about to reshare content that came up in the city classifier that sort of just gave you some words to the lies? Like, hey, make sure this content is x, y, z before posting, et cetera. Like, I'd be really curious about the consumption effects then of content that's been produced in a more thoughtful way in comparing those different AD groups. Yeah, love that comment. And I don't think anything we're suggesting precludes any of that and more innovation in terms of how these platforms are structured. So another hat I wear is working with new public. And they're trying to think through how is it that you would design social spaces in more public-friendly ways that allow people to work productively together, for example. So I don't think any of this precludes new innovation in terms of thinking through how platforms should be designed. But I think that it does say that the effects of some of these things that have been proposed as ways to reduce affective polarization, for example, they don't seem to have that effect at least under the confines of the study that we did here. And I would just add to this, one, if you look at the research on misinformation, there's been a ton of work on various interventions, including there's a whole subfield that's called nudges. And then there's also a friction subfield, which interestingly enough, if you look at Molly Roberts' work on censorship in China, like that's basically what the Chinese government does is try to use his friction for a ton of censorship. So I think there's lots of good examples to look at in that particular regard, people that focus on misinformation, but you could focus it on other things and you could focus it on higher quality content and those kind of things. And just one quick caveat on Talia's point, answer to you, like, if I was a content creator right now, like I think I'd be interested in looking at what was more likely to be reshared and what was prioritized by the algorithm for this study, but I would do it with a caveat that this is data from 2020 and these things are very live, living, breathing organisms, well, that's a bad, isn't it? But they're changing all the time. They're complex, like this was going back to the first question, there's complex interactions of lots of different features and I would not for any way shape assume that if you could tease out from something we found in 2020 that it would still be in place today. It might be, I just would go into it with like, you know, cautious eyes, which again, going back to the first question, I think points to the importance of hopefully this kind of study is not a one-shot deal that it becomes something that's repeated over time and repeated in different contexts in multi-party systems and we get, you know, we're able to design more studies that are able to be implemented in other ways. Awesome, we're a little past one, but if you guys are willing to take the last couple of questions in the room, awesome. Both of you have been mentioning really about left and right, liberal and conservative. How about so-called moderate, like, I myself consider to be a moderate and I just, at the time of election, I would depend upon what the candidate stands for. It's not the left and the right and even it happened in Massachusetts. You know, Massachusetts, we had Republican governors and even it voted for President Reagan in 1984. So, and the conservative, there can be quite religious people who are good-minded, respected, respectful to others. Then there may be people who are hating others. Still, still, they're all clubbed and medged as conservatives.  So, merging everyone as liberal or conservative. So, to me, it is somewhat not very informative. I mean, just for my understanding. And again, when you ask your users, how far they were very true to their conscience?  They might have altered their behavior just for your study. Is there a possibility? Someone altering their behavior just for your study? Great questions. Happy to make you start with the first part of it and jump into the second part as well. So, the first part of the question is about what about moderates? And across these studies, we do look at moderates as well. So, in several of the studies, we look at those people that have views, you know, 0.65 and above, 0.35 and below. And consider those in the middle to be moderates. And in our life-longed study, we did do it so that we split people right at 0.5, 0.5. But we did do some robustness checks to ensure that the results would hold if we also allowed for some moderate in the middle. So, I think it's a really good point. I think there is a moderate component to this. And as we saw in some of the work where we're looking at what sort of sources do people see in their feed, it did change the percentage of sources that people were seeing from moderate or mixed ideological sources. So, I think you're definitely onto something. And also, just to acknowledge your broader point, I think, which is, are there more axes than the left-right axes even with the middle? Yes, absolutely there are. And we just focused on that one in this study. But there are more complex ways to look at this beyond thinking about that. And then I think the second part of your question is asking whether people would have altered their behavior as a consequence of being part of this study. You know, for those people that were informed about the possibility of the study, might they have decided to change things? You know, I think there's a possibility, I can't quantify it for you necessarily, but I will say this took place over a three-month time period. We can look at how people behaved in a control group versus a treatment group where we were altering things and we do see differences in their behavior. So, maybe they both altered in some way, but we still see differences between treatment and control. And we would expect that to be randomized across the two. And you know, these were subtle changes, right? I think it would be hard for a person to all of a sudden like, oh, they removed reshares. I just think that would be exceptionally difficult for someone to have really thought through and have that conclusion in a way that they would either hypothesis guess or act against the hypothesis. But you may want to add more. Is there one last question? Yes, there's time. Very brief. Very briefly, the, so the only non-mull that popped off the slides and then he alluded to in the Q and A is the decrease engagement with political news under one of the conditions and then decreases in news knowledge. Was there any sense that there was a concomitant decrease in the intensity or salience of sort of politicization among those people, even if there was no movement in beliefs or affective polarization? Yeah, I mean, that's the reshare study, right? And so that's the one that did have this interesting finding where it made, you know, what we discovered from the first part that Talia was showing you is that people were getting less news and then not surprisingly, they knew less about the news, right? And this is pretty consistent across, there's been, my lab has done a couple of different Facebook deactivation studies, one in Bosnia, one in Cyprus, right? Oh, thanks, thanks. And in those papers, and then in the all cotton gas scale that did in the US, it's pretty consistent across those studies. This has nothing to do with this study, you take people off Facebook, they're a little happier, but they know a little bit less about what's going on, right? Like those are, there's a three studies that all find those kind of findings. And so, it's not, you know, so what we learned was that, what we may have learned is a little bit of the mechanism, but like we've learned that reshares play a role, which kind of makes sense, right? If you think that like, lots, there are some people who are more, you know, we all kind of know these, and it's gonna be a little anticodal, but like we know people who are, who go online and like to share news a lot, right? Like that's kind of the model of Twitter. And so if you get people away from the things that people are resharing, maybe you're getting more original content. And so, I think we've learned something about that, that part of the news ecosystem comes from sharing links as opposed to everyone posting them organically.  That was like, would you have lower levels of affective polarization? I mean, that was the one, you know, that was the one thing that we went in there expecting to find from everything that I've ever heard about it. So the surprising of us is these results came in too. And just adding that there is a pioneer too with respective partisan news collects that kind of relates to all of this as well, that you can find in the papers, but yeah, we didn't have a specific measure of salience in some most important problem. All right, well, thank you, Josh. Thank you, Talia. I know there's more papers coming out, so hopefully we'll have you back in a few months for an update perhaps, but thanks everyone for joining online and in person today for the awesome questions and hope to see you next week for our last installment with Ethan Zuckerman. But thanks again to Josh and Talia for today. Great, thank you so much. Thank you.


This morning, meeting with the DFAT group, they were asking me what are some of my challenges as women president, and I was jokingly saying one of my challenges is being short.  So that's number one challenge, but thank you very much for the invitation to speak this afternoon. And I join you in acknowledging the presence of the traditional leaders and the people who own this land and thank them for this opportunity to share. I'm not going to say too much, I'd rather listen or respond to questions. I'm hoping that you will have questions to raise. I want to just say a few words about my journey as a woman into this position. You know, I come from a family of ten children, and I'm number eight in the family. So obviously I wasn't a leader in the family because I was very unimportant at the very bottom of the heap. So lots of my brothers and sisters were leader of the family. So this is definitely something I have to learn as a teacher.  I consider those positions very important, you know, and I think as someone who came this far, I started learning about leadership being a mother, even though it's a small family. Still, you have to organize everything around making sure that you have enough money, making sure that you're disciplining the kids like this one, making sure you're on top of their various plans to do otherwise. And coming from the Marshall Islands, I come from a metrilineal society, as you all know. There are lots of saying about the role of women in my culture. You know, there are so much compared to roles of women in the coming men in my culture. So it goes from being the real mother for the family, the nurturer, to being the cheerleader as well for the men. So it goes the old range of roles for women. And I think for us in the Marshall Islands, women have naturally been taken on leadership role because of the cultural expectation as well. Although when we look at the political leadership, we haven't really done that well. We've had, since our constitutional government, had one woman in our parliament until last year when we have now three. This is 38 years later, we have three women in our parliament. So I'm wondering whether we'll take another 38 years before we can have another three women, so we can have six women in our parliament. So it's not easy for women to get elected, as you all know, to political office. There is the lack of organization behind women to support their running for office. As you all know, most of the political groupings are parties. We don't really have political parties in the Marshalls, but we have political grouping. And a lot of these are led by men, so if you're not part of the group, you don't necessarily get the support of the political parties behind you. There is the issue also of resources and especially self-confidence. I think these are the issues I see as impacting women's getting into a leadership position, especially in the political arena. We have to be very confident. I sometimes talk to my colleague, women friends in the Marshall Islands, to put their names out there, because during election time, inevitably very few women would put their names out there. And again, it comes back to confidence. Many women don't want to put their names out there and then lose election. They don't like that. So they want to make sure that they have a better chance before they put their names in the roles for election. So the more women put their names out there, the more they can have self-confidence, or they can take risks and just put their name out there, I think the better chance for more women to be elected. But again, the network of women is important. The support of other women is very important in getting women into a political position. In the Marshall Islands, and I don't know elsewhere, but a lot of women don't support other women running or being in leadership position. It's a lot of the time the women who are the one kind of pulling back on their friends and not helping them, not supporting them to go forward. So it makes it a little bit more difficult. I think a lot of awareness, a lot of discussion on human rights, the ability, when you think about it, it really comes down to social issues and the ability of women to take part in political discussions and decision making. It's the ability to see women as just another group and that can take part in discussion and in leadership as well. So it's human rights. Where are the rights of the half of the population? We often ask that. So when you look at that as opposed to men, for instance, women, I think it probably makes it easier for women to get elected or for. So it's not eyes against them, but it's the ability the need for women to take part in decision making role as well. So for me, I started out as a classroom teacher, as I said earlier, and going into Secretary of Education. As a Secretary of Education, I realized that the extent of your ability to make a difference is very limited because the minister, the people in cabinet make the final decisions on where the resources go. As you know, if you can put resources, if you can make a decision on where the resources go, that's where the strength is. And so I decided to run for politics because I want to be in a role where I can make a difference in terms of where the resources go in education and to be able to make a decision on how education is being managed or direction that education will take. And so that's why I first ran for politics. It wasn't to be in this position. But after four years of being Minister of Education, there were some political issues coming into the new term or the last election of our leader. And I found myself being asked to take on this role. And it just, you know, I like to say that I was at the right place at the right time because I don't think I planned it. None of my friends planned it, but it happened. And a lot of things I think in a small community like the Marshall Islands, people know each other. They know about your background. They know about the responsibilities, the seriousness, the integrity. Everything like that is open book in a small community. So taking responsibilities in my career as Secretary of Education or as Minister of Education, I was able to do a lot of change in the direction of our education in terms of creating a new policy for language, language policy for the Ministry of Education, focusing on ensuring that the native language is taught from first grade all the way up to 12th grade before the emphasis was on English. So we had to do some shifting there. Also, during my tenure as Minister of Education, we were able to pull out the public school system from the Public Service Commission so that the agency can be autonomous and everyone can hire its own staff and create their own salaries and everything that has to do with selecting your own people as well as managing their own budget. So that was a big undertaking for the Ministry of Education at the time because our budget, our constitution placed the responsibility of hiring or promoting on the Public Service Commission. So the ability to move that out of the Public Service Commission into an independent agency took a lot of lobbying and getting a lot of support from our male counterpart in Parliament and other leaders. I think the fact that I know about education a little bit and they could trust that I had some ideas about where to take education in the country, that comforted a lot of our leaders and they were able to vote with us and make that happen. So right now we're moving in, I think, in a much smoother direction and the education is moving. There's a lot of areas to improve yet. We have a long way to go, but at least it's easier for the Ministry to run its program to select the people that they need to be to work on teacher salaries and benefits as well as other issues related to education. Thank you.


Hi everyone and welcome back to another episode of our Rotorank roulette series, my name is Lee also known as Osiris and we are going to be continuing on today with this random roulette team that we kicked off with at the start of the week thanks to your good nominations as always and talking about nominations if you go over to our community section of the channel there is a post there right now asking you for nominations for next week's picks on the roulette wheel and it probably will be the last week in this series before we get back to concentrating on more serious competitive pokemon and our daily battle series of school hard knocks that we we normally are known for on the channel so that is all dependent on the rules being released this weekend at the wheel championship so it will depend on that and I will obviously let you know next week if it will be the last week or we'll continue on for one additional week which we might do but regardless if we do or not it's been a lot of fun and I hope you are all enjoying things so far so today is wednesday it is the day that we can activate those four bonus buttons that we have access to with the legend maker our random eyes to switch up and our patreon button but like I said in yesterday's episode because of the team and how well we've been doing and I am claiming that victory from yesterday so we're sitting on four wins one loss which is phenomenal for the team in a roulette series anyway um it's been going really well I'm going to actually not change anything up I don't feel like changing anything yet there's still lots of features of the team that I think we can we can definitely have a look at shenanigans definitely one of those pokemon that we haven't featured so far this week so it'd be nice to say a little bit of action from shenanigans even though I've said that and it's not the most favorite pokemon by a lot by a lot of players but it's it's still a good pokemon um especially in an unrestricted format as we've seen it do well so many times this season but without further ado let's hop into it today let's hopefully continue on this this good run that we've had going for us and as always if you do enjoy this sort of content my friends as we hop over screens and do do leave a like on the video it really does help the channel out a lot it helps the videos that helps me out helps the community out a lot and do subscribe to the channel for more pokemon content and as always leave your comments down below I love hearing from you all all the time we've got Tristan up as our first opponent oh for pokemon confident Tristan not worried about anything else so we'll get into team preview this might be a bit quick so we've got Tapa Koko, Thunderous, Incarnate, Mewtwo and Parkier I would say it's probably Mega Mewtwo um because there's not other mega options on the team Tristan the man he doesn't need any other pokemon uh Tailwind it's gonna be tricky to get off because of the priority taunt on that thunderous uh with the prankster ability there does my opponent have a way to hit Shininger I don't think they do so this is definitely one for Shininger I definitely feel like Zekrom's a good pick here as well um yeah I'm gonna bring Zekrom up front we have to worry about Koko Z move of course um I'm gonna bring Shininger I'm gonna bring Sogaleo and I'm going to bring uh Mercro because I think if we can get rid of that thunderous early on then Mercro could be good late game to get a late uh Tailwind up to kind of support everything else and Mercro is generally quite good against Mewtwo it's not bad against Parkier either so um yeah we'll go with that Paul Tristan making him wait all this time he doesn't really have much option with the team selection screen there he's got just the leads to sort out and then he's clicking in so he's probably sitting there thinking come on come on come on clicking but we're ready to go so we'll see what Tristan is going to bring out as with these four Pokemon it's not often you see teams of just four Pokemon and it normally indicates that someone's testing something to see let's see what will we get to make up those last two and it's always a good way to go around it um just play with four if you've got a core of four and then play on on showdown on battle spot whatever you want whatever your preferred format is and then see what the weaknesses are and patch them with those last two slots so we're going to see Mewtwo and thunderous come out for my opponent um did we just do a fusion bolt the fusion bolt the Mewtwo I mean they've got no way to hit the shaninja um so one of the things we could do is just we could ally switch would be quite nice wouldn't it fusion bolt the Mewtwo and just go for a cheeky ally switch might be a nice option to do because I think they're not going to target the shaninja unless they've got shadow ball on the Mewtwo which they potentially could have but Zecombe would say that pretty nicely so there's your aura sphere doesn't actually affect the shaninja and crunch into Zecombe so good job that we went for that ally switch there we got a free fusion bolt off into the Mewtwo though which is which is very nice and we do know that thunderous has access to crunch now we're going to be able to pick up the Mewtwo this next turn um with a fusion bolt but maybe it's a good idea to go after the thunderous now with a uh I'm going to go a fusion bolt into the thunderous here I'm not going to ally switch this turn I'm going to play the mind game slightly always like to do just protection ninja here uh we might take crunch into that slot my opponent might just see through that I'm going to see shadow ball here it's into thunderous though attacking themselves there's a crunch into that slot and we are going to free the Mewtwo taking down its own thunderous here just for the shaninigans yeah heck of it so maybe we are able to pick up the thunderous which we are here with Zekrom giving ourselves a nice hand up and uh taking away that one option I feel that my opponent does have to deal with that shaninja although they do have shadow ball on the Mewtwo which we'll be able to hit it for super effective damage uh and we are going to see pocket hit the field now so we've got to be careful around the the spatial rent here for sure um I really want to drop a drako big shout out to joy for that one um on the park here but do we ally switch that's the thing do we get caught up by it um I would say my opponent probably thinks we are going to ally switch I'm going to shadow sneak and I'm going to go for do we go for the drako do we call it do we say that we're going to ally switch I feel like my opponent thinks we're going to ally switch I'm going to just go for the drako into the park I'm not going to ally switch here just go with the mind games shadow sneaks probably not going to take down the Mewtwo I'll be surprised if it does yeah it's not going to it's a nice strike okay oh we're faster than oh no we missed the pocket oh come on and it actually targets us oh come on takes down that poor Zekrom a drako let us down big style um okay well me we've got the shout like yeah we're fine now you can bring in mokro we're definitely gonna I I definitely think the park here is a trick room variant so we want to try and shut that down now um we'll shadow sneak the the Mewtwo pick up the knockout there um and we'll taunt the park here because we don't want the trick room being set up um especially when we've got Sougalio on the back to come in we want to try and have that come in in good form physical form to hit everything first but I think we've taken the Mewtwo down this seals the game up for Sheninger to kind of steal this one for us so he said at the start we hadn't seen much of Sheninger this episode but it's going to be able to do a nice bit of work in this game for us so special round into mokro that you're like coming in handy there taking that pretty comfortably takes that 50 health we can tailwind this next turn if you would like to because yeah it might be worth doing especially with the type of cocoa coming in for my opponent tailwind and shadow sneak yeah shadow sneak the cocoa I'm more worried about the cocoa at this point than anything else and yeah we'll tailwind with mokro although like thinking about that taunt now it probably doesn't make much sense that my opponent goes for the trick room but it kind of deters me from tailwinding knowing that the trick room could potentially be there and knowing that the park here is slower than our zekrom makes sense that it's a wild charge to wonder god predicting maybe an ice which then rewards fear it's going to be into our mokro is this going to pick up the knockout no no so we're getting another another turn we'll just go for another shadow sneak into the cocoa go for a foul play this time into that park here and if we lose mokro now it does mean that sogaleo can come in and do some work but like I say I don't think my opponent's got a way to deal with the um the shin ninja now I don't think yeah agility type of cocoa nice nice cocoa going up to 300 speeds that now aquatail coming out into the wonder god not going to do very much but um the taunt does wear off so we'll get another turn to taunt this park here which I'm going to take the opportunity to because I do feel well even though the type of cocoa is gone for um the agility the trick room's really not very likely I don't think at this stage anyway so and shadow sneaks gonna get the jump on cocoa whatever so the next shadow sneak should get the cocoa electrical coming out will be enough to get the mokro here and parking I can be able to do anything to our wonder god shin ninja which should seal up our first game for us today but I mean we don't really need it too much now because sogaleo can come in um and what's gonna be best here really to go for uh since your strike probably gonna be the strongest like I don't really want to take the drops from super power so um we'll just since your strike it's not gonna be super effective by any means but we know our aura sphere um and special rend an aquatail of what's on the park here so it's not going to be like threatening a knockout by any means onto sogaleo here you do pick up the type of cocoa finally with ash ninja since your strike coming out into the palcare hasn't got a berry though that makes it a bit more difficult just drags it out a bit more doesn't it um we'll see we might see perfect they're not though as well do nice damage so we'll definitely get it the next turn hydro pump as well so no trick room there on the park here big big big hydro pump there and um yeah no no worries about a berry at all so we'll just shadow sneak and we'll get the knockout with the sun see a strike if the shadow sink doesn't pick up the knockout here um yeah I don't think shadow sinks can get it from this range oh it does oh she ninja so strong so good game to my opponent and a nice one to kick us off with today so five wins unheard of one loss can we make it six one going into halfway through the week that means that we probably yeah six one with four games left can't finish negative if we win the next one so it's gonna be good unless we play like extra games like we did last week and it comes back to bite us in the butt pretty hard like it did on Friday was heartbreaking but um we'll continue on as we'll always do in the series and oh we've got to do this we've got to do it with this next win if we get it we are going to be back of our 1500 for the first time I think in this entire series can we do it I'm excited I'm like like I'm excited that we can actually do this we can get over 1500 like the last time we were over it didn't really count because we were streaming midweek and it bumped us up because we were running a solid team we were running your irons team on stream so we had all those wins got right back in the the mix of things this could be the the the actual the the authentic result that we were looking for here today uh to take us over the 1500 mark hopefully it doesn't take too long to find the next opponent um I'm just gonna cut right now and we'll come back when we do bump into them and we've got our next opponent of the episode we've got king 15 15 rating player and playing quite yeah it looks like a nice team let's get into it so king playing the team of zekrom kaelga whimsicott tapakoko salamence and gengar we've got two potential megas on his team here between that salamence and the gengar then you've got the tapakoko for the train support to sport that zekrom got the tailwind support uh speed control from the whimsicott also on that salamence and then the kaelga which is probably going to be primal kaelga here now does my opponent have anything yeah one like the megas if we can get rid of the megas then I think shinninger has a really easy time so shinninger is another pokemon I want to try and bring here potentially to help us out against this potential uh team uh because once gengar and salamence are gone then it does free up the um the shinninger to kind of do some decent work at least so I will bring it in the back potentially though I think of anything else it's a nice switch into kaelga at the very least um well that's do I want to bring do I want to bring pz pz is not a bad option to bring up top but then the gengar does cause us a few issues so it might be better off bringing something like zekrom here then shinninger in the back um and so glaleo so glaleo is not bad here to be honest um so I'm gonna yeah that's gonna be my fall let's see if we can do it here we go friends and if we're gonna win here I'm not changing the team tomorrow there's gonna be no activation of buttons at all tomorrow so we'll see and even if we do lose this one I still would be hesitant whether I want to activate any buttons or not let me know in the comment section definitely let me know what your thoughts are if you want to see buttons activated you don't want to see buttons activated it'll be really good to hear we're gonna see salamence and the wednesday call come out for my opponent and that's a horrible shiny salamence I'm really not a fan of shiny salamence looks so much better in that nice blue color but it's just personal preference I'm sure there's a lot of you out there that do like like salamence here um so terrible activating we could drake a meter this salamence straight up if we wanted to um and tailwind whether I make or at least then uh yeah I'm gonna do dark type we can't be taunted by the the wednesday call um yeah and I think the only thing we have to fear here is if this salamence has drake a meter of its own you potentially could we're not gonna see any switches uh I think if we were like to the top of coca could be the worst one um we'll get the tail wind up with our macro what we're gonna see this whimsical go for you see a double up moonblast we're going into zekrom oh into mercro no okay we get a tail wind up it's fine um although oh no that's the one thing we didn't want to see that's terrible for us that's terrible for us okay um I'm gonna have to bring in so go now sorry my phone keeps beeping it's um I'm getting a call it's my mom she doesn't realize I'm recording right now mom I'm sorry I'll call you back I'll call you back stop ringing me I can't turn it off um all right so I will taunt the whimsical because I feel like we need to try and and shut that thing down uh hopefully we can taunt it before you can do anything to us and I'll go for a sensual strike into the salamence ah salamence like they do occasionally have to echo meter in this format to deal with things like rick wazza um okay we're gonna see zekrom come in I don't mind this too much um I'm really at all has a taunt so at least we're gonna be able to shut down this whimsical it's not gonna be able to get its tail wind up yeah it's going for it so that's fine it can only throw a moonblast which it does threaten mokro 100% um but a foul play should be able to get the zekrom I think after a sense just probably not enough but we can earthquake we can earthquake and foul play um and that's probably what I'm gonna do go foul play um and I'm gonna earthquake because then we break a potential sash on the whimsical if it stays in uh mokros obviously a flying type whimsical can take down mokro as well uh that's a big thing but I want to be able to take down uh the zekrom here that's the big thing that's the most important thing for us so we've got to try and take down the zekrom yeah which gets a nice damage onto it damage onto the when me there's a foul play this will take down the zekrom get rid of that threat now we'll probably lose mokro to a moonblast I would imagine here from the whimsical so we get rid of mokro it's done a decent amount of work and then we've got what we've got on the back peasy no shininja oh okay things just got better except that the the salamence is pretty tricky to deal with so we need to yeah we need to deal with the salamence post haste um yeah it's gonna go probably hyper voice I reckon you could toxic the salamence makes it a bit easier since you'll strike I want to get rid of the whimsical like as soon as possible because I don't want it that taunt wearing off and getting access to tailwind because potentially kyoga's in the back right and that thing comes in with a tailwind behind it and it's going to be very difficult um probably like at the same time you're thinking well get rid of the salamence because it's the thing that does threaten shininja but we do have white god that we can utilize with shininja as well uh with that subalail so if it's only got hyper voice to uh to utilize maybe it has maybe it hasn't got double edge maybe it's got drake me a hyper voice tailwind protect could be it's set I don't know it has got the hyper voice here it's going to take us down to our sash with uh shininja not going to do very much to our subalail though that's a big thing first we have toxic toxic as well so uh the white god support will help like immensely against this this kyoga that I'm kind of suspecting coming in now um kyoga so we can toxic the kyoga my god I think that's not a bad thing like one of the things we could see is potentially my opponent going tailwind um yeah tailwind protect kyoga potentially no wonder if we can win just by white god toxic stall that would be an impressive way to win but we will throw out a white god now my opponent's locked in pretty quickly so there's a white god for ourselves I'm going to see hyper voice yep what's about oh come on we needed that to hit we really did like just to start the clock like ticking down we really needed that to kind of tick in we did 100 I'm going to protect shininja because the thing is it once we get rid of the salamence you've got to think the salamence is gone then the kyoga cannot beat the shininja so the salamence is like the key to this game for us we get rid of that we win this game doesn't matter what what what happens after that that's that's the big the key for us now hopefully my opponent thinks I'm not going to use the spread attacks this turn this this gives us the open it's all the mind game from here really um it's quite easy for them just to sit and think well I'm just going to click these moves again because if you like if you don't white god I'll punish you and that's the that's the big thing double edge beautiful what's about not so evil hopefully oh hanging on by the skin of his teeth wow okay right now we need this to do enough damage for the toxic to tick over and take it down that that is what we need come on so go leo do it do it for me man that's not I don't know I don't think it is gonna shadow sneak will take it down there now shadow sneak will get it so we will shadow sneak and we will white guard because I don't want the kyoga getting a free kill on a so go leo hopefully shadow sneak is enough to get this salamence you you've got to imagine it is from this point the toxic's coming in handy the toxic would have been handy against the kyoga for sure but then uh kyoga cannot beat cannot beat shinninger so once this salamence is gone it's gone and it takes us that will take us to six one all going well of course it's not over yet so there's the white guard to block those those nasty spread attacks that we don't like there's shadow schnick schnick schnick oh shinninger the boy here's the mvp today people are gonna hate me for this but uh there's a wonder guard not gonna do much um unfortunately uh but yep now we can just toxic the kyoga and um go for that uh did we just super power super power man yeah just get some damage on the kyoga and there's the forfeit so we do pick up the victory come on victorious six one six one it's not happened before we've never been this this had these these good results before in the roulette series and it's great because we're coming to close of the series sitting at six one we cannot finish negative now this week we are going to finish positive and it will take us over 1500 as well which is like it's everything all encapsulated into one I'm so excited I'm so happy they were finally here and I knew it would pay off eventually and I uh I just want to dedicate it all to you lovely people out there that's what I'm gonna do um but we are gonna end it there guys we're gonna wrap it up we'll be back tomorrow I'm seriously thinking we don't need bonus buttons we might do it Friday I'm not gonna do it tomorrow so we'll be back with more episodes tomorrow there will be more games tomorrow so do stay tuned see if we can continue this on I mean imagine if we could finish the week with like one loss or week that would be incredible wouldn't it it would be insane if we can do that have a great rest of your day though thanks as always for tuning in I hope you have enjoyed it and we'll see you for the next one guys until then take care and bye bye