All right, we're gonna get started. Thank you everyone who's joining in person and online for the first RSM speaker series installment of the semester. We're really excited to welcome John Lucas Frajini from Boston University. But before John Lucas takes the floor, he's gonna be introduced by Joe Walther, one of our visiting scholars this term. Joe is the Bergelstein Presidential Chair in Technology and Society at UCSV. He's also a distinguished professor of communication and the former director at the Center for Information Technology and Society. And with that, Joe. Thank you, Nick. And thank you also, Nick, for arranging this speaker series and providing me for John Lucas to come today. I'm a fan of John Lucas's training. He does computer science work that I can understand, and that's saying a lot. But John Lucas and his colleague, they've been looking at a lot of the things that can serve us online in terms of hate speech, misinformation, conspiracy and so forth. And it's not hard to find an instance of any of these things. But so, Luke and his colleagues have uncovered the process of this, how the actors responsible for this move across domains and move across platforms and what some of the content is that they move and coordinate in their work. When I discovered their work, it blew my mind. It was really nice, solid, evident, illustrated in really fascinating ways in the papers that they write to just help blow open some of the social dynamics that otherwise are only represented by static instances of hate messages and so forth. So I'm impressed in the big fan of his work. And then I asked John Lucas for his beta. Now I don't like him anymore, because I'm jealous. He has recently tenured as an associate professor. He has scads of publications and conference proceedings and articles. He has millions and millions of dollars in grants. He's got anything that we would ever want an academic to do. But beyond that, as I said, his work is meaningful and exciting and detailed and precise and tells us a whole lot. So thanks for coming today. And thanks John Lucas for joining us to tell us about your research. Thank you. Thank you so much for the kind introduction and good morning or good afternoon, everybody. It's great to be here. So as Joe said, I'm a computer scientist. So what I do, I develop computational techniques to better identify and understand malicious online activity. So I started looking at malware, online fraud, spam, this kind of automated bad activities that we're all familiar with. More recently, I was drawn into this rabbit hole, if you like, of online hate and conspiracy theories and misinformation and so on. So today I want to give you an overview of some work I've been doing with my students and my collaborators on better uncovering what's going on out there. And to start, I want to give you an example of something that happened to me a couple of years ago now. So one day I was sitting in my office and I started receiving weird emails that look completely random. So the first one was this email where somebody was asking me for advice on how to cope with a recent wave of hate and bigotry and antisemitism and all of that. This was early 2017, just to put it into context. Okay, kind of random, but I had published some papers on the topic. So it might make sense. Then I started receiving other emails. So this one was asking me about help on some class project for a class called Community Studies Project 12, which I had no idea what it was. Okay, weird. And then I started receiving hate emails, wishing me all sorts of bad things and so on. And this was in a short timeframe. So what was going on here? Well, later on I discovered that someone posted this screenshot on Fortune, on the political incorrect board on Fortune. And this was, you know, an anonymized blurred out screenshot of a class assignment where a professor was asking students to go on the community and start posting about controversial topics, you know, gender issues, LGBT issues, women's rights, all these kind of things. And if you are at all familiar with the community, you know, that's a very much out-fried community. So they were not pleased with this. And they also posted this screenshot, they blurred out the email address down there, but they didn't blur it out completely. So you can see G at something.edu. So they interpolated it in the thread and they got convinced that it was me for whatever reason. So they found an old email address of mine that sort of fit in there. They mistyped it, but that doesn't matter. And so they decided to go and attend. So, you know, they put out some of my public information, they called for a text, and then they started posting templates of emails to send it. So this was the email that I received, the first email I showed was posted as an example on Fortune and I later received the same exact email, eventually they figured out it wasn't me and they stopped, okay? But this gives you an idea of what these coordinated harassment attacks look like and what we call a raid. And, you know, as you can imagine, if I receive this kind of attack, as a white male professor, whatever, you can imagine how much other types of demographics might be affected by this. And after that, you know, this kind of blew up. Now, if we hear about it every day, we hear about, you know, celebrities and gamers, all sorts of people being attacked online. And so we set to study this phenomenon in our work, and we found that it's quite common. So in our limited view of, you know, the political incorrect board, we looked at attacks targeted against YouTube, specifically YouTube videos, we find that it happens about two to five times a day, which, you know, it's quite a lot, but it's only one community. It's fairly small, comparatively speaking, to, you know, other platforms and so forth. So what are we going to see today? I want to give you an overview of four projects that I did. So the first one is these coordinated attacks against YouTube videos. So somebody posts a link to a YouTube video, but for whatever reason they don't like, you know, it might be against what they believe, they might want to harass the person and so on. And basically they invite people to storm and disrupt the video in the comments. The second one is about Zoom Bombing. So that's a similar threat that kind of emerged during the COVID-19 pandemic.  And then I'll talk about some other work I did on automatically identifying online risks in private conversations online. So cyberbullying and sexual solicitation and things like that. And finally, I will conclude with the work that we did in looking at what happens after communities get the platform because the most natural way of dealing with online hate and conspiracy theories and whatever is just suspending those accounts, suspending those communities and so on. But these are real people so they will move somewhere else and what happens then? That's something that we need to keep in mind. So to begin understanding what we are talking about, let me tell you a little bit about 4chan, which is the first community that we studied. So 4chan is an image board and so people can post an image that starts a thread and then they can discuss it. It's organized under a bunch of topics, various levels of weirdness and toxicity and so on. What we focus is the political incorrect board, which is one of the most popular communities that creates havoc on the internet. They did things like turning Microsoft's chat bot to a racist a few years ago. They organize our source of raids. They claim that they sort of were extremely influential in getting President Trump elected through these kind of meme wars and so on and so forth. What's interesting about 4chan compared to other communities out there is that at any point in time, there can be only a certain number of threads active. And so every time a new thread is created, a thread has to die basically. It's archived. And after it's archived, it goes into this archive for a week and then it's deleted. So content is also ephemeral. And also users are anonymous, there are no accounts so anybody can post and there is no way of telling who's posting what. And as you can imagine, this combination of anonymity and ephemerality sort of encourages all sorts of bad behavior, especially when there is no moderation from the administrators and so on. So just to give you a sense, we find that the medium lifetime of a thread on 4chan is 47 minutes, so they are fairly short-lived. And we started collecting data from 4chan. We're still collecting it. So this is from a couple of years ago, we had over 134 million posts. So it's fairly large, not as large as Twitter or Facebook or Instagram or whatever, but it's still very large. We're still collecting this kind of data. And then we started to focus on this coordinated paid for harassment campaigns. And the way this works, as the example I showed you at the beginning, somebody will post a thread calling for an attack and then people will get together and carry out the attack. So what they might do is doxing the person, finding more information about them, social media and those or even worse, private information and so on. And then they will go and execute the attack.  And ultimately cause harm. And in this process, many dynamics emerge. So they will post evidence of what they did. They will say, did you see that? Did you see how they reacted? They will congratulate each other and so on and so forth. So how do we automatically trace this? So we decided to focus on YouTube videos because we found that it was the most popular domain linked on the platform. That's true for any social media, probably YouTube is the most popular content out there. And we found examples of attacks called against YouTube videos. So they will post a link to the video and then people are supposed to go and attack the poster in the comments. And if you remember, threads on Fortune are ephemeral, so they have a lifetime. So what we were interested in understanding first was can we see an effect on the YouTube comments after a link is posted on Fortune? So if a link is posted on Fortune, we expect people to click on it, go and comment on the YouTube video. And since the thread is ephemeral, that makes our life somewhat easier because we can sort of see between zero and one, that's the lifetime of the thread. How many comments do we see? Do we see a spike in comments? And we do. But that's not telling us much because that's what we expect to see in any social media platform, right? Once somebody posts a link somewhere, we expect the reason why they're posting it is for people to go and visit it and engage with it and so on and so forth. So we needed something slightly more sophisticated. And so we resorted to a signal processing, essentially. So the idea there is that, how can we simply model this behavior of having a synchronization thread where people coordinate and then the YouTube video were actually posting comments. And cross-correlation is this metric that is used to model how synchronized two signals are. So what we did, we modeled the comments on Fortune and the comments on YouTube as signals as a train of impulses and whatever. And well, we use this mathematical technique to find out how synchronized they are. Ideally, somebody will post a hateful comment on YouTube and then immediately go back and report on the thread. This is what I did and this is what happened. In reality, it's a little extra than that, but you get the idea. And this is interesting because it gives us a number, which is the lag between the two signals, but then we can use later on to automatically identify videos that we believe were red. And what you see in this plot is basically, we have a synchronization here, so zero indicates perfect synchronization.  So we ran the comments through the hate base API, which is an API for identify hate speech. And so the dots in red contain hate, the ones in blue do not contain hate. And what you can see is that basically, the smaller the lag, the more synchronized the fortune thread and the YouTube comments are, the more hate we see. And if there is really no synchronization, there is really not much hate. So that allows us to extract a set of videos that have been targeted just by looking at this number. We don't even need to look at the comments, we don't need to train classifiers or large language models these days to identify what's hateful and what's not and so on and so forth. So that's fairly accurate and quick to do. And then by looking at this set of videos, well, we made some observations. So we find that, as I said, people like to report back. So they will talk about what they did, they will talk about what happened in the attack, potentially how it affected the victim. So some of these videos were actually live streams. So this could be done in real time instead of just being a video posted there to be watched later. We see that they overwhelmingly target vulnerable demographic. So we see attacks called against women, against members of the LGBTQ community, against minorities, and so on and so forth. And then in some cases, we see that even if the attack does not start as targeted, so it's not calling for an attack against the specific demographics and so on, the commenters will go after other commenters who may be part of a certain minority. So sort of the attack becomes targeted as it goes. In an opportunistic way, if you like. We see a lot of concern trolling, which is sort of interesting. So these would be attackers who pretend to be real concerned citizens about, sort of what I showed you at the beginning, when the person pretended to be concerned about racism and so on and so forth. And they try to get the victim into either contradicting themselves or saying something controversial or whatever, or just wasting their time. And why is that interesting? Because it's very difficult for moderators to identify this kind of behavior. It looks legitimate, it's just not. And what do we do about it? It's not necessarily against the terms of service, it's a hate speech. Actually probably it looks exactly like real people being concerned about an issue, at least on the surface. And then we also find that this is a trolling community. So people troll each other all the time. So oftentimes they don't act upon this stress. They say, we are not your personal army, NYPA is something you often see. So we don't want to participate into this attack, go away and so on. So it's kind of a multifaceted landscaping. And then we want to understand what can we do about it? So we have this set of videos that were attacked and clearly YouTube has a hard time moderating them. Because we all know that content moderators are a scarce resource, they're overwhelmed. There is so much content being generated all the time that where do we even start? So we developed a system that basically given a video extracts information about the video. And so this would be the title of the video, how long is it, what are the tags? We look at specific keywords, cops, woman, whatever. We look at images. So we look at the thumbnail of the video. So we have a deep learning model that can tell us what is in the thumbnail. And then we extract the audio and we look for keywords in the audio as well. And the idea is can we take all the signals and combine them to identify videos that are likely to be moderated, to be attacked. So if that works, that could be an additional signal for YouTube or whichever platform you want to protect. Telling content moderators, this is a video that you should pay particular attention to because it's likely to be attacked in the future. So this could be a prioritization system that could be used by content moderators. And it works reasonably well. Any questions so far? Yep. On the previous slide, the bottom statistic about the AC of 0.94. Yeah. Have you applied that forward looking and then seen if the videos you predict will be rated R rated or is that purely retrospective? This was purely retrospective, yes. Because it wasn't easy to extract this ground truth of videos that will be attacked. Yeah, but that's a good point, yes. Also it might overfeed, so it might be biased on videos that were attacked by that specific community. So there might be other communities that target other types of videos. The second question, can we predict which videos we have in example, whatever is shown on the screen is theoretical or hypothetical, do you have a specific example? I'm shorting so much. No, it says that people are up there, people online and they can hear you, even if you don't shot. Yes, yes, yes. We have examples of videos, so this would be typically videos about social justice or related topics that would get specifically targeted. I think you are giving only a hypothetical. Just give a very specific example wherein the YouTube removed the particular item. Oh, where they removed, they don't remove the particular item but they would remove the hate speech. So what we're looking at is really hate speech in the comments. So again, I keep asking the same thing, do you have a particular example? Yes, yes, yes.  Yeah, okay. I have a good question. My question is related to the, maybe we'll enter this in a later stage of the talk with a higher strategy of the prioritization of the content itself and not necessarily the attack. Is that because of the space in which the hate and harm happens that this prioritization is particularly useful so that moderators are more out in the look but it seems that it's kind of like a remediative strategy instead of a preventive run, right? It would be a remediative strategy but it would allow moderators to look at the risky comments in a more timely fashion, right? Because the problem with content moderation is that there is so much content out there that it takes a while to act upon bad content especially with online harassment and so on. Once something is posted, if you don't get it very quickly, the harm is done. Sure, YouTube goes back and removes these comments but if they do it days later, it's not really very helpful, yeah. So then I wanted to talk about, oh yeah. I have a question about scale. Yeah, have you looked at the breath of this trolling raid activity from 4chan to YouTube relative to the entire sort of inter-platform type of practices that are going on across the breath of 4chan and also relative to any, I wouldn't say non-normative kind of indirect cross-platform behaviors but maybe even pro-social kind of behavior. So it's relative to the whole scale of cross-platform activity and the pro-social side of cross-platform activity. How does this sort of trolling raid-like behavior scale? It is a minority. I want to say among all the videos we saw on 4chan, it would be in the order of 1% or something, the ones that are actually attacked. So most videos are posted for whatever, this is a funny video or whatever it might be. Most of the inter-platform activity is news. So people will post news articles and then there is some inter-platform activity because people will go and post comments. We did some work starting the hateful comments on news articles. We find that often news articles are taken out of context and just posted on 4chan to make a point, especially for immigration issues or things like that. Yeah. So now you say it's the management of YouTube or Facebook. Even government agencies, they try to keep track of such issues, right? Yeah. And again, even for them, it should be a very major task and they'll be able to fully solve the problem. Yes, especially if you are doing it in a cross-platform fashion, there is so much data and using this kind of techniques but just look at synchronization might be more robust than looking at text that people can try and change the way they speak to avoid being detected and so on, but it's a huge problem. Yeah, definitely. So yeah, then after doing this work, we realized that a similar problem was happening on Zoom. So we saw news reports and so on or even in our own online classes, they will get disrupted by trolls. And we found a similar pattern of these Zoom links being posted on 4chan. We also looked at Twitter, so there will be posted on Twitter as well with an invitation of going and disrupting the meeting. Okay, and the difference there compared to what I talked before was that these are real-time events. So you cannot attack a Zoom meeting after the Zoom meeting has ended and you cannot, yeah, so it has a smaller window of opportunity. And the other thing is that we can see links being posted on 4chan or on Twitter for Zoom meetings but we cannot really see what's going on on the Zoom meeting. While on YouTube, we could go and check the comments. So we applied a slightly different technique here. So we had basically multiple annotators go and check the threads to tell if it was a call for an attack or not so that then we could characterize what was going on there. And we found 134 calls for Zoom bombing over a period of a month, if I remember correctly. So it was, that was the beginning of the pandemic. So it was a problem that was kind of surging. And what did we find here? Well, we found that the huge majority of attacks here are calling to disrupt online classes, online lectures. So it would be high school classes or college classes. Everybody went online, so it was very common and nobody knew how to secure these things. The second interesting thing, we found that 70% of these attacks were called by insiders. So it would be legitimate students in the class asking strangers to disrupt their own classes because they were bored. And then we also found that most of them are targeting meetings that are happening in real time. So there was wording in the thread saying, my class is happening right now, come and disrupt it. So these were not really planned ahead of time. And all these are interesting because when defenses for Zoom meetings were developed, they were assuming a completely different fact model. So they were assuming that somebody from outside would come in and disrupt the meeting. So they would tell you, set up passwords, but if your attacker is an insider, they can just give the password to someone else. They would tell us, set up a waiting room so that you can tell who's real and who's not and keep out attackers. But actually we found many instances of students saying, oh, here is a list of the names in the class. So you should adopt the name of some other student so that the teacher wouldn't be able to tell who's an attacker or who's not. We see that oftentimes the attackers are taking the name of the host so that the participants are confused if they don't know what's going on and so on and so forth. So after a while, services started developing like one time unique links for each participants but this is not very common and it's also typically a paid option that they offer. So I don't know if it's still the case, but for Zoom you could only do that if you had an enterprise account. And so that wouldn't protect, later on we talk to many people from volunteering organizations or who would have volunteer ran Zoom meetings or seminars and so on. It wouldn't be an option for them to adopt this kind of defense. And other area that I looked at which is sort of similar in spirit. So we were looking at private conversations on Instagram. So everything I talked about so far were not the Zoom bombing, but the YouTube stuff happens in public. So we wanted to understand what happens in the private space. What kind of risks and malicious activity are users victims of? And so here this is part of a large NSF project with several other institutions. So what we did, we set up a data donation for Instagram. So our participants in particular teenagers and young adults could donate their data to us and they could also label the data themselves among what did they think was safe, what did they think was unsafe, what made them feel uncomfortable and so on. Which was interesting because there is a lot of results showing that if you ask external participants or external annotators who are not the victims themselves or are not in the loop, what they end up labeling doesn't really match the experience of the victims. And so we had over 28,000 chats with millions of messages. Some of these chats were labeled as safe or unsafe. So we wanted to understand what do unsafe conversations look like. And so the first thing we found is that just by looking at again metadata timing information and so on, we can tell with fairly high accuracy whether a conversation is gonna be risky or not. And why? Because the participants tend to disengage. So if they start being bullied or they start receiving solicitation or something that they don't like, they will stop responding. If they are being bullied by multiple people in group chats, there will be this sort of one way stream of messages. And why is that interesting? Because that potentially allows us to develop detection systems that don't really go and look into the content of conversation. So they are more privacy friendly. They could be applied when there is end to end encryption. So we cannot see the content but we still want to do something to protect participants. And so we developed again a set of classifiers. So we looked at metadata. So everything I said, but doesn't look into content. And then we also looked at text and we look at images and so on. And what we found is that, as I said, the metadata classifier is fairly accurate by itself. So potentially it could be used at least as a strong indicator that something wrong is going on. Then obviously using all classifiers looking into text and so on is more accurate. And then we also found that if we wanted to identify the specific type of risk, so we have several types in our work, in a bullying, sexual solicitation, hate speech. We also look at spam and things like that. Then we need to look at content. Yes. What sort of metadata were you looking at? So it would be timing of the messages. It would be the length of the conversation. Sometimes the participants will specify the relationship with the other party. If it was a stranger, a friend, an acquaintance, a partner or whatever, some information like that. So yeah, if you want to look at the type of risk, we need to look at all of that. But this sort of gives us a direction to look into. Maybe this metadata features are promising to be adopted in a scenario when we want to guarantee more privacy to the parties, which is what we all agree is good. And most messaging platforms are going in that direction. So we have end-to-end encryption and so on. Finally, I want to conclude with this other project in which we looked at these challenges that deal with the platforming specifically. So coming from computer security, we used to work on anti-spam, anti-fraud, anti-malware, whatever, and so the solution was always the block content, delete content, and the problem will go away. In these cases, it's different because we're not dealing with automated programs, we're dealing with real people who can reorganize, go somewhere else, potentially go somewhere else, out of sight, and so there can be, they can get radicalized more, those communities are not moderated, and so on and so forth. So we looked at two communities that were the platform from Reddit at different points in time. One was the Donald, which became infamous for sharing conspiracy theories and all sorts of disinformation and so on. At some point in time, in 2019, it was quarantined, so people could still access it, but they would receive a warning, and so that's what said the administrators off, saying, oh, they're after us, they're gonna ban us, let's create our own community, the Donald.win on their own website, and let's migrate there. And then later on, the community was actually banned on Reddit, and so people moved to this new community. And the second one is our incels who are the involuntary salivates who've been linked to misogyny and acts of violence and mass shootings and whatnot. They were also banned, and they moved to their own forum. So we wanted to understand what happens to these new communities that are now somewhere else, and they are not on a mainstream website, so there's probably no moderation or very lax moderation and so on. So, yes? If they are not on the mainstream website, then they are effecting this, not that significant, right, because for people like me, I'm not going to find that in general. Yes, but they could coordinate and come back to YouTube, for example, to mount attacks, or they could, you know, if there is a link between online hate and offline violence, they could organize acts of violence offline. So they themselves interact? Yes, and then they can go, if they are building a conspiracy theory, then they can go and push it to other communities. Because I was comparing the situation with money swindlers, you know, a company cheats people of money.  Yes, yes, yes, yes, yes. But here, you know, the web is all interconnected, so they can still come back in other forms. So what we did, we did a regression discontinuity analysis, so this is an example from the Donald. We have the same thing happening on incels, so the line in the middle is when the migration happened. And what we find, first of all, not everybody migrates, so the number of users on the new community decreases. And why is that? Well, likely because Reddit is a general purpose community, they have so many subreddits and people kind of, you know, various interests. So they might be tangentially interested in whatever the Donald is talking about, but not enough to, you know, create a new account on another community or go and check it and so on. So only the people who are really committed would go. But among these people, we find that in general, their activity increases, so they become more active, yes. Hi, my name is Glen. So this part where you said the othering, is it through a lexical content analysis, looking at words like they and them? Yes, yes. So we get into that, we find that when we start talking about we a lot more about them and so on, we find that hate speech in general increases. So yeah, not only their activity increases, but potentially the community becomes more insular and, you know, gets even more polarized, because now there is only the people who are, you know, like-minded, there is no content moderation and so on and so forth, yes. Yeah, I have a question about, so could we assume that people who choose to migrate are not, like, exactly comparable to those who stay? Then could we assume their trend is continuous that enables us to use RDD? To use RDD? Yeah. Because I assume that people who are migrate, who choose to migrate, they are more dedicated to this topic or more polarized. So can they really qualify as a counterfactual of others? Well, not necessarily, but what we wanted to understand here is how does the community change, right? So now the community will only be made of extreme people or committed people and will be outside of the moderation capabilities of RDD in this case. So what we wanted to understand here is, you know, what happens next and, you know, this opens up many research directions on understanding, you know, is the platforming of these communities in this shape and form the right way to go, you know, probably the toxicity of RDD decreased overall. So if that was the goal of RDD's administrators, you know, they succeeded, but overall as a society, these people didn't go away, you know, they just moved out of sight. So that's what we need to keep in mind. So to conclude, you know, show the number of examples where computational methods can help study and identify coordinated online aggression. And I pointed a few directions in which, you know, these kind of techniques could help platforms develop better mitigations. But once we do that, we should also keep in mind this potential unintended consequences of, you know, just if you ban people or remove content, it doesn't necessarily go away, it just goes somewhere else. And with this, I thank you. And I'm open to questions. Thanks very much. My name is Adam. We're in a blue stripe shirt. I'm questioning about the Instagram private messages piece of your research. And I appreciate you providing all the detail about where you derived your initial data set. But given that it's explicitly about a tool that you've developed to work on private messaging, what do you see as the use case for whatever it is you've developed? I mean, I can spin it out, but I would love to hear what you think because it's not like Facebook can use it to monitor what they could, but we don't want it to private messages, you know. So where does that better prediction come into any sort of utilitarian play? Yeah, that's a good point. I think in general there is always a tension and a discussion whether platforms should read your private messages and look for illegal content or whatever or protect their users and so on. And in general privacy is obviously a big concern. So that's why people are advocating to move to end-to-end encryption and so on. So what we wanted to understand there is can there still be some indicators that can be used without looking into what is being said or even who we are talking to and so on, right? Okay. So in the data collection piece for the same study, I know you've had users volunteer the private messages and you did that analysis. Was there a strategy in approaching the platforms to see if they would use some of the data that you would need to do as a researcher or was it more of a strategy? Yes, so... What's their response on it? It's always tricky and in general, getting user data from platforms, especially if it's not public data, it's something they will not do. And that's one of the reasons why we had this volunteer base. Approach in addition to the fact that we could, they could label their content and tell us what was risky and what wasn't because if we did it ourselves, probably we would end up with a very different definition and then maybe whatever we developed wouldn't really work. Yeah. So I wanted to look at the discourse raised here in your conclusion to ask a few questions. By effective mitigations are you only interested around the content moderation practices because there is an argument that has been made that there could be another form of mitigation around amplification and incentivization. Yeah. But that would then mean looking at different sets of practices that may be more positive or more pro-social. Have you considered any of that type of... So once you identify this content, you can do various things. So I have work that looks into this information space that looks at soft moderation. So you don't remove the content but you add a warning label. So that is one thing. There are these shadow banning or demoting content, not showing it in recommendations and so on. That's something that could be plugged into this kind of systems. So instead of deleting the content or removing the content, you just don't show it to somebody or you use it as an input to your recommendation algorithm and suddenly that content is not recommended anymore, et cetera. Sorry, just to follow up, I'm just wondering if there's any means with which there are other ways to incentivize platforms around doing non-less negative, less bullying, less presidential work. Yeah, that would be great. It's very difficult to get feedback from them and also see what happened with Twitter in the last few months. I don't think any content moderation went away or at least from what we can see. So yeah, obviously it would be great to have a more safe environment. I'm not sure where the priorities lie really for different stakeholders and so on. We have a question from Augusta online. Did you study the growth of the new communities over time, in particular active recruiting on mainstream platforms with LinkedIn, YouTube video comments, et cetera, and if they grew, did they stay more toxic or have more othering? We didn't look into that specifically. For this study, those communities grew in general in the sense that the dot-win type of community attracted more of the platform communities. So it became a conglomerate of communities instead of just being the Donald. So that we observed, yes. Interesting that you raised... Are you okay? Yeah. I was actually one of the council members on Twitter since responding and then I resigned and must actually instigate a harassment against me and the two women that they fight over the else. So this thing too, for me, I like what you're doing here and also I'm working on some similar methods. I do that a lot because we're in the US and it's very US-centric technologies that we're always looking at the content. So I like some of the work that has been done elsewhere where it's like actors, behavior and content ABC and the focus is overwhelmingly on content and what I was really looking at, say, in literature is known as amplifiers like influences, right? And so the categories of actors and how we detect who they are and whether they impact, like, for example, swarming and what does that mean? And so I like to have a discussion afterwards too about actors and behaviors because I feel that that needs to be looked at more and then the impact in terms of the First Amendment and what we can do about it. Yes. And because we're at a law school, I've been looking at Laura's last six words and what he was saying about replica behavior and how that pertains to the First Amendment. And he was saying that if technology changes so much and impacts democracy, then we should do something about it. And so for me, this is the missing bit of the conversation that I feel is a suicidal effect, but it's so siloed that we actually may not look at it in a very integrated way. Jack, thanks. Thanks, Rebecca. I do have a couple of questions. The first is on the slide that you showed us that showed various stages on the chat. Over in the far right, you had one individual who was a victim. And your examples about when you and yourself were victimized, that was personal and individual. Yeah. When these YouTube attacks, the YouTube rates take place, then these are more or less public postings rather than private postings, correct? So I wonder if you have a sense of proportion about that. Are the rates typically for posting in public spaces or do they frequently involve the doxing and finding a way to disrupt the individual within their own private communication space? I think that public ones are more common because they're easier to carry out. It's easier to find public information about people. It's easier to get traction. It's easier to show that you did something because it's out there now. So they both happen, but if I had to, I don't have hard numbers, but if I had to guess the public ones happen more often, yes. It takes more preparation and more orchestration to truly dox someone and go after their private communications and so on. Thank you. One more question, please. You've mentioned how and when you've spoken before, particularly in regard to Zoom bombing, that one of the comments that the Zoom bombing post tend to be racist, they tend to be misogynistic, but it was your impression that those comments were not born out of prejudice and enmity, but were rather the most disruptive things that somebody could come up with to say. Yes. I wonder if you could elaborate on that. Yes. So we had some follow-up work that's not published yet, but we spoke with victims of Zoom bombing and several people told us that they're not in the mind of these attackers, but their feeling was that they just joined to disrupt the meeting and then they found the most low-hanging fruits. So they started harassing people of color or if a speaker was a woman or whatever they could find. But that was not the original goal of the attack. We didn't target the meeting because the speaker was a certain person or anything like that. Turning back to the YouTube breaks, then, which you said, for example, as a social justice video was likely targeted. Do you think it's the same activity going on? In other words, do you think the culture war is really being fought on YouTube in these grades that people who are very opposed, very ideologically opposite what's in these videos are doing these attacks or are they just really easy sitting ducks for major disruption? So do you think they have a different character, in other words, than the Zoom problem? I think so. I think that at least the way we saw it, for the videos, they typically would talk about what the video is about. So there would be a more targeted purpose. For the Zoom bombing, it was just that all these high schoolers were bored, at least early in the pandemic and they wanted some thrill. Yeah. Thank you. I think we have time for one last question. My answer is a comment that Joe's question, the last question on the YouTube race, this is the Zoom bombing. I would assume what you just said too, because during COVID, it was boredom on the behalf of those people taking part in Zoom, such as maybe high schools, but also more than that too, right? But YouTube race, I think, would be characterized by presence of amplifiers, influencers who are coordinating it by doing a mock video. I'm mocking you. I'm using coded language, and my followers are not going to harass you. So I think there is another link there that there is someone who's openly then mocking a video to make that actually happen, whereas Zoom bombing is more like everybody and then do it together. There's no one head honchos. Yeah, that's a good point. On four-chance specifically, since everybody is anonymous, there is sort of this high mind or whatever of these attacks, but the same rates can happen in a number of platforms, yes. So there could be the influencer directing their followers to harass someone, and yeah, that happens too. All right, well, thank you everybody, and one last round of applause for John Lutka. Thank you. Thank you so much for joining online, and if there's more food outside, I think we want to help themselves to more lunch. But thank you, everybody. Have a great day.


The earth is the art. The photographer is only a witness. Jan Arthas Vidran Creative Earth Institute of Photography


Hi everybody. Thank you for coming to my talk. I'm extremely nervous. I'm excited. So this talk is about MySQL and Kubernetes and my name is Patrick Galbraith. Something about me, I've had a long lasting relationship with MySQL. I've worked on MySQL for MySQL, various other companies working on MySQL. I've contributed to MySQL. I worked on various MySQL projects, Memcache, UDFs, DVD MySQL. I'm a pro driver. I still am a co-maintainer of that. Federated storage engine. I contributed the original example of running Galera on Kubernetes in the examples directory for Kubernetes. So I love learning languages, whether it's programming languages or spoken languages. I love being with my family outdoors. I love Chile. It's the best place I've ever been. And as you can see by this page, I'm not a designer. So I started working for this great company, Oracle Dine in September. And I'm having a great time. What we provide, and most of you probably already know this, is DNS. We provide DNS for some of the world's largest and most admired web properties. We have the largest, fastest, and most resilient DNS networks in the world. Over 3,500 enterprises, brands like Netflix, Twitter, LinkedIn. We also provide email for organizations with large sending needs. We also are building a cloud, Oracle Cloud Infrastructure, and Dine provides edge services for that cloud. And we're hiring. This is on the slide, so please do take a look. So what is this presentation about? Well, it's about running MySQL on Kubernetes. And some of those things, databases are extremely complex pieces of software. And Kubernetes actually makes it easier to run databases. We're going to talk about stateful applications, which the database is, a complex stateful application. And in that, we're going to talk about stateful sets. We're going to showcase this with Vitess by talking about Vitess and doing a demo. We're also going to talk about operators. And I'm going to be very excited to talk about the MySQL operator that Oracle has been developing and give a demo of that. With time permitting, I have some extra slides. I went crazy. I had 60 slides at one point. Somebody said this is a two-day seminar. So that's why I'm speaking with some cadence here. And please do save questions to the end. So it's a tale of two open source projects. We have MySQL, which has been in software years, been around for 20 years. That's quite a while. And it certainly is ubiquitous and everywhere and used so many places. And there are a lot of variations of MySQL permutations. I heard even heard somebody was telling me about the work that Alibaba has done. I know friends at Facebook, they've made their customization. And then there's Kubernetes, the Linux of the cloud, as we heard it at this conference. I got to say the fastest growing open source project. It's application deployment done right. Definitely community driven. Look at all the people here. 4,000 people. That's fantastic. And it's a moving target, rapidly developing. When I started this preparing for this talk, I had an understanding of when I had put together the original Galera example and things have changed.  So with databases, it's a question of pets versus cattle. A database containers tend to be more pets than cattle. That seems obvious. You care greatly about them, like your pet. A pod of multiple databases or instances or one. The description of pod is we get to run a bunch of containers on one node. Well, we don't necessarily want that with databases. And the documentation it says, Kubernetes pods are mortal. Mortality is not something we necessarily like to see as an attribute in databases. A database is a complex stateful application. We want consistent access. We always want the same database to be there. Safety don't scale if it's unhealthy. That's a bad thing. And of course, persistent storage, which is kind of an obvious thing for that. So here's some ingredients. I like this picture of chats. I love chats. They're delicious. And so we have some delicious ingredients with Kubernetes. We have services which I'm not going to talk about what that is. Everybody knows what that is. Persistent volumes. Persistent volume claims and storage classes. And then of course, we have a newer resource on the block, stateful sets. And then within that there, we can talk about init containers if we have time volume claim templates. And then there are operators which allow you to extend Kubernetes and they utilize custom resource definition. Then we have sidecar containers to do some handy tasks for us. And node selector, I was going to talk about this. I probably won't have time, but it's definitely something to know about for databases if you want to run a database on a particular node. So first of all, we're going to talk about stateful sets. So stateful sets provide guarantees about ordering and uniqueness of pod resources and pods themselves and the way that they're named. They maintain persistent sticky identity for each of the pods across rescheduling. It also ensures ordinality. When you run pods within a stateful set, it's going to be zero through whatever number you're going to run and they're going to be named that way. They're also interchangeable and you won't run a pod with the same name. You won't have two of the same thing. They have a stable network identity. First of all, the name of them is whatever the service name is dash ordinal of how many are ever being run and then you create a headless service. If we have time, we can discuss this and that is the domain name for that pod and that's how you would access it. They have stable storage and stateful resources. Pods remount the same storage through restarts. So when there's a restart, it's the same thing as it was before. Safety, we don't scale if it's unhealthy. So we now have a, I'm going to show these two illustrations. They're really good in kind of bringing the point home as to what the differences are. So with a deployment replica set, they're not started in any specific order. The name is, you have a name and then you have a numeric value of the deployment or whatever idea there is and then you have a random alpha numeric and that changes from, for each pod, they're all different. It's not fun to try to log and figure out what containers are running because you have to look and see what they're called. And it'll still scale if there's a crash and it'll replace with another non-unique name. But with a stateful set what we get, we're started in order. You don't have to sit there and say, I want to run this one, this one, this one, like I did with my original example. The names are unique and distinct and it must replace, if there's a failed node, it's going to replace that same node and it's going to be that same node for all intents and purposes. And you don't scale the cluster when it's unhealthy. So then we have operators. It's an application-specific domain controller. It encodes application-specific domain knowledge through extending the Kubernetes API. What does this mean? This means we as operators, we do certain things. If I run the Galera cluster, I do it in a certain order. I maintain it in a certain way. With operators, you build that into it. You build that into the custom resource definition when you create your operator. It enables users to create, configure and manage stateful applications because you can add that functionality in there. And it builds upon resources and controller concepts and it leverages Kubernetes primitives like replica sets, stateful sets and services and other resources. An operator executes common application tasks. You design it to run whatever you want within your application and it does it for you and hides all the gory details. It's all installed as a deployment. Once it's running, you utilize it. And you run it with custom resource definition types. Some of these here, you see at CD cluster, Prometheus and MySQL cluster and MySQL backup, which I'm excited to talk about. So I like this graphic. This is one I copped from Corus's website where they discuss operators, but it's really nice. And I saw this morning in the keynote, there was something just like this. What an operator does is it observes the state of what you have running. And in this example, it shows, oh, I only have two running. Oh, and one of them happens to be 3.1.1. I really want 3.2.10. And that's where the analysis kicks in. Analyze, and it says it should be the version 3.2.10. And there should be three members. So now we're going to act. We're going to bring up that missing node. And we're going to also upgrade. So this graphic here, it was more about me wanting to talk about stateful sets, but it works well with operators as well. This is an example Anthony A did of a simple MySQL replication. And what it utilizes is you have a simple pod with two init containers. The first one is init MySQL. It sets up the configuration file for MySQL, whether it's a master or slave. And then there's clone MySQL. What it does is it receives a backup from the extra backup. If it's a master, it doesn't do that. And it does this based off of it being ordinal number zero. And then there are the app containers, with MySQL obviously. And then there's an extra backup container. Extra backup. And what it does is once MySQL is up and running as it starts a backup so that the next subsequent node that's running can receive the backup through the init container. And why do I have this up here? Well, there's some complexity to this. And this is the kind of thing that you would put into your operator. And that's what's really cool about operators. So the MySQL operator we have arrived at this point. I always wanted to do this is one of these things that I thought, oh, I'd be great to do this. Well, I happen to work at Oracle now and we have a group in Bristol who's developed this. And it's scheduled for release early 2018. You can deploy a highly available clustered MySQL instance onto Kubernetes with a single command. It watches the API server for custom resource definitions related to MySQL, MySQL cluster, MySQL backup, backup and restore are made simple with the MySQL backup resource. Utilizes persistent volumes, persistent volume claims. Under the hood it utilizes group replication. And you can have automated backup to restore to from different object storage, scheduled and on demand. So MySQL operator, some of the features, I've already mentioned some of them, but this is a nice graphic. You can see the operator here is doing all of the tasks with the MySQL cluster you would normally do as an operator. And the Kubernetes API is through custom resource definitions has it run those things. It provides configuration for your MySQL cluster with simple Kubernetes objects like secrets and config maps and backups. You can define what kind of backups you want, what the policies are to make sure your data is always protected on demand or scheduled, as I mentioned. Here's an example of... I'm trying to see where we are in time. So here's an example of creating a cluster. We define our persistent volume in its file here. And then we have the MySQL cluster that we define. We say we want three of them. We want to give it a name of example MySQL cluster with volume. We use a volume claim template. I love these. These are great. They just happen to work. And you say what storage class name you want and you submit it and it creates a cluster. And then with a backup, same simple situation, you have your cluster, MySQL-test cluster, and then you specify that's what cluster you want a backup. You say what your credentials are, OCI is our object storage, just like S3, and you specify that that's what you want a backup. And it backs it up. And the commands are... You use kubectl to create it, both your backup and your cluster. You can also do kubectl get MySQL backups to see what backups you have, and then you can get a specific backup. You can also describe backup. I don't have that in there, but you'll see that in the example. So now we're going to run a demo, drum roll. So the initial state, nothing is running? No, I can't. That's the way it was recorded, and I didn't do that particular recording. I tried getting as big as possible. Pardon? Come on, plus, plus. Okay. Nope. Okay. So we've created the operator, and now I got to focus on this. So now we're going to tail the logs. We should see that we have the operator running. We see that it's running, a replica set. So at this point, we can now utilize the operator. We're going to tail the logs of the operator and see what's going on. It's kind of... I like the way this was recorded that you can see this. Now we're going to go into the cluster manifest file. We're going to do three replicas. We're going to call it MySQL test cluster, kubectl create-f, the name of that. And at that point, you can see the logs that it's starting to bring up the cluster. Now we're going to watch the pods as they come up, observe each of the nodes coming up, and also go in the kubedash and watch this as well. So we see the first one starting to come up. And also we have a secret in here that's been created. This is how the database, you'll talk to the database once it's up and running. We now see that the first one and the second one are now running. You can see the messages. The third one is now starting. You see inside of kubedash that it's almost up. The third node is up. So at this point, we should have a running cluster. So now we're going to go into the first node, and we're going to connect to the database. We're going to find out what the database password is from the environment. We're going to use the MySQL iNoDB cluster made a data schema, and then we're going to find out what the group membership is from hosts. And there we have a three node cluster. That was easy, wasn't it? So the next thing I'm going to show you is a backup. This is even easier. So initially we have nothing in our object store. We have a running cluster. Now we have some credentials and a bucket name. We're going to look at the backup manifest, make sure it's what we want to back up. First thing we're going to do is we're going to actually put some data in there, and make sure that we have some kind of database to refer to when we back things up, create a simple table, put a row in there, just pretend it has billions of records in there. We're specifying what database we want to back up and the credentials. Now we submit that backup. We create that backup, I should say. Backup is scheduled. Backup created. We have a list of backups. Now this backup is a MySQL dump at this time, but we're adding functionality for enterprise backup and extra backup. That'll be configurable. Now we're going to describe the backup. And we see backup.file.target.ez and we should see it in our object store. Yes, it's there, isn't that great? That's exactly what we want to see for ease of backups. So now we're going to talk about the test. The test is sharding on a silver platter. I know it's not a silver platter, but I love the album. It doesn't have anything to do with my age. It's a database solution for deploying and scaling and managing large clusters of MySQL instances. It has built-in sharding. It's been YouTube's database store since 2011. It's what Slack uses. It looks like a MySQL database in terms of usage, but underneath it can be sharded in any way that you want it to be. It's cloud ready. The cloud is coming, as Sugu would say. It helps you scale with transparent dynamic sharding and the ability to continue sharding in or out, depending on what you want to do. It also has cluster management tools for backups, shards, and schema management. It talks MySQL client protocol, GRPC, and GRPC. It provides connection pooling. These are GO connection pool that talks to a small MySQL connection pool. It protects MySQL with query deduping, rewriting, sanitization, putting limits on unbounded causes, and a number of other things. There are monitoring tools that are built into all of the test binaries. If you want to migrate your MySQL application to the cloud, the test is an excellent vehicle to use. So here are the components of the test. We have, first of all, your application talks to VT gate. It's a lightweight proxy used for routing queries to the appropriate VT tablet. And it also reassembles the result sets from the different shards. So VT gate then talks to VT tablet. VT tablet pause or comprised of VT tablet itself, which itself is a proxy in MySQL. VT tablet is a proxy that does all of that query sanitization and blacklisting, deduping, putting limit on unbounded where it causes. Then next we have the topology server. Topology server is essentially a service, a lens in front of at CD, and all of the data, the coordination data, the key space graph used to determine where the shards, which shards you use, it's also used for initial discovery of what's there. And then you have VT CTLD that provides a web interface and it does all the cluster management work. And then another piece in here that I added that's not on the graphic on the website is orchestrator. Orchestrator pertains to reparenting and it talks to VT CTLD and through that it does the reparenting when that needs to occur. And then there's VT CTLD, that's the command line for interacting with VTES through the topology server. So sharding. A shard consists of, as I mentioned, a single master that handles rights and one or more slaves that handle reads. You can split horizontally, you can split or merge shards in a sharded key space. You can also do vertical sharding, moving tables from an unchartered key space to a different key space. You can do resharding, split, merge, add new cells and replicas. It does this through filtered replication using GTID, which is the key ingredient to ensuring that your data goes where you expect to go. There's also the concept of V schema. This is a VTES schema. This is essentially the schema for how you shard. Think of it that way. It ties together all the databases.  And then there's V index. This is a cross shard index that provides a way to map what, in your table, what would call them to your key space ID. And I'll show you an example of this. So right here we have, you know, on the left we have our regular user table schema definition. And then on the right we have our V schema. We list what V indexes we have. The first one is called hash, and it's of type hash. Then we say what tables the user, and we say what column we're going to shard by, and what the V index name is. And it happens to be hash, as we just saw up there. And this is what you submit to VTES, and that's how VTES knows how to shard your data. There's also the concept of reparenting. We can change our shards master tablet to another host. We can change a slave tablet to a different master. As I mentioned, GTID is the magic in all of this. There are different types of parenting. There's a planned and emergency. It kind of speaks for itself as to what it is. It also provides resharding. You can split, merge, or add new cells or replicas. And I've said this several times, but filtered replication is the key to this with the GTID. Backups. Backups are really simple with VTES. And there are a number of plugins for different types of object store, or NFS, Ceph, Gluster FS, I believe. And the thing to know is VTTablet has to have access to where you're backing up to, and use VTCTL to run these backup commands. And when a backup happens, minuscule instance is stopped and it copies the data. That's how it does it. And this works just fine because we have the master to still continue writing to. We also have restore. The VT tablet has started with the arguments of where the data is. And restore is made. And then here's an example of some of the commands you can look at in your spare time when you look at my slides. So now we get to the fun part. We're going to do a demo. And what we're going to do here is we're going to create two shards. We're going to load a database schema and vschema, scale one shard from two replicas to three. Then we're going to delete the stateful set. We're going to pull the carpet from the rug from under its feet, and then delete the master pod and then see VTES do some magic of reparenting. This demo is made possible. It's a demo by Anthony Yeh. You've heard his wonderful talk this morning, his keynote. And this is where you can go play with it yourself. So now what we're going to do to run VTES is we're going to install the Helm chart. There should be nothing here with VTGate status or VTCTLD. Helm chart is installed. So once this is installed, we should start being able to look at our stateful set and then see what pods are coming up. The zone one dash main. Those are the two shards. We should watch them come up. There'll be some errors. This is a bug. I won't go into this. This is a known issue. It's nothing to worry about. So now we're going to watch things come up. The initial state is there's no replication running. We have not set that up. Things are still coming up, as you can see in cube dash.  The graphs are showing things happening. So now we're going to initialize the shards. We're going to initialize both shards. We're going to apply a vschema. We're going to apply a regular schema and then we're going to back up both. So now you'll see replication get started and a master and a replica set up. You can see here it's already happened with the dash 80 shard. And now with the other shard it's coming up as well. Both shards should be ready at this point. We won't see any traffic on it yet. We're not running anything against it. That's going to change soon. No traffic. It's nice to have graphs there. You don't see anything in the query log either. There's a nice query log UI there that's built in to VTCTLD. I get that messed up. I'm a tongue twister here. So now we're loading. The script is loaded. We now have a master and a replica. We're doing a backup right now and it's not serving while we do that. We're going to watch the logs a little bit to see what's going on. The backup's almost complete. The status is backup. Right now that means it's not serving. Now the backup's complete and it is serving. It's now ready to use. So now the manuscript is completed and we're going to run a load test against it. This is really fun to watch because you get to see the graphs. Load test is now running. We now see traffic showing up against both shards. We're going to look at the master traffic and see what's going on there. We see a different, various statements running against the database. Write statements. Also going to look that, you know, we never did do this when we were initializing it, but we're going to look at that the schema is actually there. We're going to look at the query log, the query log UI. You can see all the different queries running against the master at this point. All write queries. Write statements, I mean. You see a color code in the commits. That's really nice. We're going to look at the traffic on the slave. All selects as you would expect. Look at the query log. All selects. Now we're going to see how many stateful sets we have. We're going to scale this one of the shards, 80x shard to three. We're going to watch the new replica come up. It's now scaled to three. We're going to look in the UI. It now has three replicas. They should be ready. Right now it's doing a restore, and the restore is getting completed, and now it should be serving traffic. The load test was running, so it immediately starts serving traffic. You can see in cube dash that it's becoming even more busy with the load test running. So now we have deleted the stateful set, and we have deleted the master. So this is where we get to see Vitess do some really interesting things here. We confirm that the master has been terminated. Now we're going to see promotion, and now what was once the replica is now the master. All without pain. Now we're going to tear it all down, because the demo is going to be over soon. So that's the end of the Vitess demo. What time is it? 4.21? So we have about, what is it? About four minutes for questions. I have these extra slides. Please, when you look at these slides, please look at collect beats. This is a great thing I wanted to talk about. There's a friend, Vijay Samuel, who works for eBay, who's written this. I wanted to cover it, because I wanted to talk more about monitoring. And I also have some other MySQL installation examples, because I wanted to do simple. I wanted to work up from simple to more complex, and so I have these in here. I have demos recorded for them, and I'd be glad to take questions now or later. Oh, there's a thank you. Thank you, Anthony. A extremely helpful, Sugu. The team I work for at Dine, Oane Lewis, who worked with his team to write the operator. Steve Lerner, who's in the audience, several nines who has a great Galera example. But anyway, questions? Yes. Unfortunately, I don't have much experience in that. A lot of what I've done has been kind of... I've worked with a lot of teams. I worked at HP and the Advanced Technology Group, and we did a lot of prototyping. And right now, I'm kind of in the same kind of role, so I don't have experience in that. But there might be people in the audience who are more than welcome to chime in on that having run it. I mean, whoever might be running Vitesse in the audience. Anthony Ye, or anybody else. But we'll take another question since I'm sorry I don't know the answer to that. Yes. Well, it's group replication, so it's not a simple master slave replication. It's a cluster. Yes. Yeah, I haven't used a lot of group replication, but it really is... It's not that simple type of master slave replication. Yes, you can. There's some of the same issues you have with Galera where you have to think about if you're going to have cross node locks, that situation, so pick one of them out to write to. I don't know. The person sitting next to you can answer that much better than I can. Probably. He liked the operator video when I showed it to him the first time. His talk was really cool this morning. Any other questions? Thank you very much for coming. I really appreciate it.