Good morning. My name is Leila Zia. I'm one of the general chairs of the web conference 2019 head of research as we at Wikimedia Foundation and Co-president of web for good which Organizes this conference I hope you all enjoyed the past few days if you were with us this morning We are going to have a keynote presentation Before introducing our keynote speaker. I just want to say a few scheduled logistical points about the rest of today Today's Friday We are going to start with the keynote We're going to go to coffee break the first coffee break of the day will happen as usual downstairs So just go one floor down in the exhibitor space Then we are going to have the research tracks and in parallel to that we are going to have Designing an ethical web, which is the special track of the day We're going to have lunch. It's usual lunch today is sunny. We started with Sun on Monday We're going to end with hopefully Sun on Friday. Enjoy the lunch outside And then when we come back from from lunch again as usual posters and demos, they're going to be downstairs in the expo hall Parallel to that we have designing an ethical web running in garden a B Coffee break will happen Same time, but you're not going to go to Pacific concourse expo hall. So just come here outside Grand Ballroom and the coffee breaks will take place here because at that time expo hall, they're going to wrap it up And then we will come back to this room at 4 p.m. There's going to be a panel And then the closing ceremony and awards for best papers best posters Actually, I am reminded that there is one thing else happening right after this our keynote today Which is in this same room We are going to ask all of you to come to the center of the room as much as possible and as close as possible to the stage We are going to have a group photo. So please help with that Cooperate with us at that point. We are gonna have we stop at 10 and we're gonna have literally five minutes to make it work So help us as much as you can you're gonna be seating on the chairs But we will just need you to come center forward And if you have red line yards, please make sure you are not in the photo or take off your red line yard before joining us in the photo Okay, so with this I am delighted to introduce our keynote speaker today Claire Wardle is our keynote. She is a Ted fellow She leads strategic direction and research at first data. She's the co-founder of eye witness media hub before first draft Claire was the director of Director of research at Tau Center for digital journalism at Columbia University She has years of experience in the area of journalism She was the head of social media for United Nations refugee agency We are delighted to have Claire today because over the past days we have been talking about what it takes to have an open web a web That is trustworthy that is balanced and that can represent the people that are behind this web their cultures their information and their knowledge and Claire is going to talk with us today about a Participatory participatory project that she's putting together with a group of people Which is going to do exactly that so The project is aiming to bring an information comments create an information comments that Represents the world that we live in not only a minority or of this world or Only part of the knowledge of this world But also an information comments that is reliable and that is trustworthy And it holds accountable to those who create the knowledge and disseminate the knowledge in in disinformation comments With this introduction, please help me in welcoming Claire Wardle. Good morning everyone. Thank you so much for coming It's a real pleasure to be here and having the opportunity to talk to you and thank you to Layla and the organizing committee For giving me this opportunity and so as ladies said, I'm going to talk about enlisting the public to build a healthier information commons But I want to start here, which is imagine 9-eleven if it happened today I'm sure you recall this iconic image of crowds of people simply standing and staring Watching in horror as the smoke billowed out of the towers But imagine that same scene if everyone was holding a smartphone. We can imagine what that would look like today So if it happened today technology would change everything rumors and misinformation would be speeding across the social networks People would be live streaming from the top of the towers Hate filled messages aimed at the Muslim community would be everywhere The outrage amplified by algorithms responding to unprecedented levels of shares comments and likes and foreign agents would amplify the division Driving wedges between different communities. I Know that you know this you're probably the most sophisticated audience I've ever spoken to when it comes to these issues But I wanted to start with an example to underline how much has changed in such a short amount of time The changes to our information ecosystem have been incredibly rapid and we Governments tech companies and civil society are now looking for fixes that can match the speed of this change But that's leading us on some dangerous paths and yet more unintended consequences And that is what I want to focus on today How can we take a deep breath and think about ways we can adapt to our new reality rather than thinking we can solve anything? In less than two decades We've seen the explosive growth of mobile technology the collapsing costs of mobile data the dominance of a handful of corporations that govern unprecedented amounts of speech globally and the decline of gatekeepers News outlets are struggling financially But more critically the days of loyal audiences have disappeared as people turn to algorithmic feeds and search results to show Them the news of the day and these shifts as you know have had major Implications the scale of which we can't even get our heads around and certainly not our hands The fact that these corporations are private means that we just can't access data in order to properly understand this new reality They have so much impact on our public sphere yet. We are powerless to really understand how it's working So when we look at the information commons, it's broken We proudly live in the information age yet the central currency of this age is no longer considered entirely trustworthy or credible We like to say we live in the information age yet Who here hasn't received a text message from a friend saying sorry to bother you It's just that crazy uncle Bob yet again has posted something in the family Facebook messenger group And I don't know if it's true or not. Can you help? We can't be proud of that reality in 2019 and while an alien visiting from Mars might be forgiven for thinking that Misinformation started the day after the US presidential election in November of 2016 Those who were monitoring information on social media had seen this coming particularly those working internationally My background I've been working with newsrooms to figure out how to verify content on the social web I helped develop the BBC's online training course about verifying material on the social web and around 2009 we were Basically establishing techniques for how to verify the video of Nader who was killed in Iran We actually were working on the rumors around Hurricane Sandy in 2012 Verifying chemical weapons attacks in Syria in 2013 the Philippines election in May 2016 and more personally the Brexit referendum the month afterwards There is no doubt however that while this is not new it's happening at a scale that's difficult to even imagine Let alone start thinking about interventions ten years ago when I was training those BBC journalists I was training them to be wary of this guy Tomasso di Benedetta an Italian who liked to impersonate famous people on Twitter This was all we had to worry about back then or I was warning BBC journalists about Wikipedia edits that might mean they Described a music festival as starting its life with the names wanky balls festival So this was actually a newspaper the independent a well-known newspaper in the UK and it got hoaxed by a Wikipedia edit Seriously, these were blissful times in comparison to where we are today We're facing unprecedented challenges connected to information quality, but before we go any further Let's just have a quick chat about language our language is so horribly muddled when it comes to talking about these challenges That's a polite way of me saying why on earth are we still using the term f asterix asterix asterix news It's extraordinarily unhelpful covering multiple things that are just not the same lies rumors hoaxes misinformation conspiracies propaganda and more importantly We have to stop using a phrase that's been co-opted by politicians right around the world from the left and the right Used as a weapon to attack a free and independent press because we need our professional news media now more than ever And besides most of this content doesn't even mask a Raiders news It appears as memes videos and social posts too much of this conversation is focused on text and URLs And yes, this is part of the problem But our ability to pass text and analyze it through NLP has disproportionately skewed our research and solutions to text Images memes and videos not just deep fakes or what we should be worried about and Through the work that we did at first draft monitoring misinformation in Brazil and Nigeria I'm also now obsessed with audio messages again Not deep fakes just voice memos that people leave via WhatsApp old school rumors left by your friend and reshared Audio the most authentic communication medium available Someone you trust is telling you a story and it's likely that you'll believe it and most of it is not fake It's misleading We fixate on what's true or false, but my biggest concern is actually the weaponization of context Because the most effective disinformation has always been that which has a kernel of truth to it Back in 2017 frustrated by the term f asterix asterix asterix news I began because it was being used to describe everything and nothing I worked out this typology as a starting point for the first misinformation at MIT lab in January 2017 I did this on a Sunday in my pyjamas It was based on the work I'd been doing for the decades before most of which had been about videos and images and the typology reflects that I Wanted people to understand that these problems were complex and there was a spectrum from satire that of course is not Misinformation when created but is increasingly used by agents of disinformation around the world Knowing that it will be shared and after a while people won't see it as satire. They'll start to believe it Another reason to label something satire is so that fact checkers don't touch it And this is an actual technique that we see we're seeing happening globally At the under end of the scale we have the truly 100% fabricated content But really the most concerning content is in the middle that which is misleading As a side note any designers in the room, please make friends with other people this was actually my first attempt at the slide and I Somebody tweeted it from miss info corny got picked up by Brian stelter and appeared on CNN with this horrific background So I just wanted a shout-out to designers that we need you more than ever I then wrote a report with Hossain Derek Shan where we published this spend diagram Explaining the difference between miss dis and mal information and we looked at two factors falsity and harm So misinformation is something that's false and could cause harm But the person sharing that content doesn't realize it's false So my mum resharing this shark photo during hurricane is an example of misinformation Disinformation is also false But the person creating or sharing realizes it's false and is sharing it to cause harm Russian operatives creating events on Facebook that are both pro and anti immigration are examples of this We also created the term mal information used to describe genuine information But the sharing of this information is designed to cause harm This might be the leaking of emails or sharing personal images of a sexual nature otherwise known as revenge porn We also talked about the different elements trying to break down the agents messages and how they were interpreted Some agencies some agents are motivated by financial gain Some by political influence either domestic or foreign and some are motivated by social or psychological factors They just want to cause mischief or even trouble and mostly they just want to feel connected to a wider group Finally we talked about the different phases of information disorder to highlight how dynamic this space is Whoever is creating a message and their motivation shifts as it's published and redistributed often with new framing messages So I'll be honest. It was basically an 80 page report that said this is hard and complex But here's some typologies and diagrams to make us all feel better But we do have to grab on to this complexity I'm increasingly comparing this to pollution and thinking about this as a health context Many different types many different harms global in scale and influence with something this complex Terminality terminology matters granularity matters. We all have to agree on what we're actually talking about So let's think about the word harm. We use the word harm But what do we mean when we say that full facts a UK fact-checking organization came out with their pagoda of harms They wanted us to recognize that the fact that the shark on the highway exists It might be annoying, but it's not harming anyone that is distinct from the kind of dangerous speech that has risk to life Either because it's radicalizing people or it's causing a public health crisis as we currently have with vaccinations globally We're disproportionately focused on political diffs information Particularly in the US, but can we start focusing our attention in places where we can see concrete will real world harms so Getting the language bit out of the way. I want to provide a foundation of why this is so complex I think that there are five interlocking issues and then I'm going to suggest how we as a wider community might start working on these Challenges by bringing in the public So first we don't have a rational relationship to information. We have an emotional one It's simply not true that more facts will make everything. Okay The algorithms that the platforms use to determine what content we see are designed to reward our emotional responses When people are fearful which they increasingly are over simplified narratives conspiratorial explanations and messages that demonize others become far more effective and Since the business model for so many of these platforms is attached to attention the algorithms will always be skewed towards emotion I know it's tempting for our community Researchers journalists and technologists to think about more nutritional labels flags and credibility scores. That's what we need But we're in this mess because people connect with information that supports and reinforces their identity When we share on Facebook Twitter and it Pinterest, it's about performing our identity The public nature of these sites means that we have to recognize this element People want to share content that reinforces an idea about the type of person they want to seem to be Most of the speech we're discussing is legal I'm not talking here about child sexual abuse imagery or content that incites violence It can be perfectly legal to post an outright lie People talk all the time about taking down problematic or harmful content But no one can agree on clear definitions and that includes Mark Zuckerberg who recently called for global regulations to help moderate speech My fear and it is a real fear is that right now Governments around the world are making hasty policy decisions that could trigger other more serious long-term consequences for our speech Here's a dynamic map of government interventions to date that pointer manages is updated by the amazing Daniel Funky Who you should really follow if you care about these issues and how they're affecting people globally already? There are almost 50 of these around the world from threats to very problematic legislation It's easy to nod when you hear someone talk about problematic content as if it's clear But I want us to look at some examples and remember that the real threat is the weaponization of context Here's an example from London from March 2017 a tweet that circulated in the aftermath of a terrorist incident on Westminster Bridge This is a genuine image not fake It was shared widely using Islamophobic framing with a number of hashtags including ban Islam The woman in the photograph was interviewed afterwards and she explained that she was utterly traumatized on the phone to a loved one and Out of respect not looking at the victim Now if you were moderating Twitter on that day, would you take it down or would you leave it up? My gut reaction my emotional reaction is to take this down. I hate the framing of this image But freedom of expression is a human right if we start taking down speech that makes us uncomfortable We're in trouble and while this example might seem clear cut to you and it does to me makes me feel very uncomfortable Most speech isn't and the lines are incredibly hard to draw a well-meaning decision by one person is outright censorship to the next What we now know is that this account Texas Lone Star was part of a Russian disinformation campaign one that has since been shut down Would that change your view? Well, it would change mine because now it's part of a coordinated scheme to sow discord Here's another example that a former colleague Cameron Hickey flagged to me Let's read it together for a second Illegal aliens are far more likely to commit federal crimes based on statistics They are 7% of the population yet they commit 72% of drug possession 33% of money laundering and so on and so on If you're from the left, you're probably automatically thinking that this is false If you're from the right, you may be thinking great. When are we going to build that wall? These statistics are factually correct The issue is the word Federal most US crime is state crime. It becomes federal when a border is involved So these numbers with context make more sense But if you fact check this it would need a true stamp over the top We don't have a way of communicating this type of nuance effectively Here's another. This is a genuine video from the lead-up to the midterms in the US a Woman went into a voting booth and when she tried to click a candidate's name the other name popped up She had she had explained what happened They took the machine out of service and she was able to vote essentially no real harm happened But this was pushed through conspiracy networks including QAnon networks and as a result This was this was pushed very very widely with this framing of you can't vote You can't trust the voting system Facebook took it down immediately saying it was connected to election integrity Twitter took longer stating it was a genuine video once they realized the coordinated piece of the campaign That had been on earth. They took it down But I want to use these examples just to show how difficult this is and finally To Brazil where we worked on a collaborative journalism project there called comprover and we had a central WhatsApp number Which over 250,000 messages were sent to us from Brazilians asking whether something was true or false This was the most shared image that we received and it's a genuine photo of ballots being moved to the location There's nothing untoward although. It's not great that it's held on the back of a truck But the rumor that circulated with this image on WhatsApp was that it had been pre flagged for her dad Bolsonaro's opponent And so one of the reasons I show these examples is because I worry that too many people believe that artificial intelligence Is going to solve these issues and I think we can all agree that we're a long way away from having AI Sophisticated enough to make sense of social visual posts like this and while we're talking about AI allow me a couple of minutes to discuss deep fakes Yes, they're scary, but honestly I'm much more worried about the kind of effective misleading content I've just shown you and also things like this This was the Jim Acosta video where a minor edit to speed up the video and to remove a spoken apology Made the CNN correspondent look like he was acting aggressively towards an aid This was pushed out by the White House. It wasn't a deep fake. It was most certainly a shallow fake and This video we've seen pop up in five different elections around the world It purports to be an immigrant beating up a nurse This video is when it happened during the French election and on the right This is when it appeared in the Spanish election and Maldito Bullo fact-checked it showing that the original video was actually from Russia Or this CCTV CCTV footage which was incorrectly shared after a terrorist incident in Brussels It doesn't have to be sophisticated by the time it appears on a small phone screen any nuance goes out of the window And if it reinforces your existing fears prejudices or worldview you're much more likely to believe it And yes, there is a worry that a supposed deep fake of a candidate emerges the night before an election and swings things But that risk already exists So yes, we have to be prepared for AI powered synthetic media But we also have to remember what tools we already have We have television archives that include older speeches and public appearances to cross check whether someone really did say something at a given time We have amazing technology built by researchers to automatically assess blood flow So you know that this video of Barack Obama isn't true or the blockchain technology newsrooms are now working on so they can protect their own Visual imagery. I believe we're still four years out from a deep fake that could really fool us And my hope is that by that time will have our own weapons to counter the threat Because I'll tell you there's nothing like watching what amazing journalists and human rights investigators can do when it comes to Verifying something like a YouTube video purporting to show a chemical weapons attack in Syria when there are no journalists on the ground You watch them cross-reference evidence from doctors and eyewitnesses on WhatsApp geolocate images using satellite Image and shadow analysis. They verify the distance of a bomb using audio forensics We're already doing some of the most incredible verification of videos And if you're interested this example from the New York Times is just one of them So next time someone tries to terrify you about deep fakes Say calm and say we're on it because the fear mongering is actually causing the real harm It's the liars dividend the fact that if a piece of audio comes out about a candidate They can already dismiss it as a deep fake. It's this liars dividend. We should really be worried about not the technology itself Even if we could figure out what to keep up or take down We've never had so much speech Every second people around the world upload millions of pieces of content in multiple languages drawing on thousands of different cultural contexts We just don't have effective systems for moderating content at this scale whether powered by humans or by technology And I'm sure many of you have seen this video But when you see these Chinese click farms you recognize how difficult it is to even Handle this kind of content at the scale that we're working on Yes, we see these in China and Russia and Iran and Saudi Arabia So fourth humans have a dark side few in Silicon Valley ever anticipated that mass shooters would live stream their massacres Very few newsrooms thought about how they would be targeted and manipulate manipulated to amplify the flawed ideologies driving some of these shooters if you considered how the Fundamental architecture of Facebook the ability to target ads would be weaponized by Russian trolls or that cybergs in some of our Porous countries would be paid to hit publish thousands of times a day to emulate botlight behavior and few foresaw conspiracy theorists building sizable audiences around fringe beliefs on YouTube and few predicted that beautiful images of lattes and sunrises Would end up losing out to outrage memes on Instagram But why would anybody think about the dark side of human nature? Silicon Valley believed that their technologies would counteract hate connection would drive understanding and therefore tolerance There's no one factor that has caused these challenges yet. Some of them are technical. Yes But much of this is driven by social political and economic factors We too often fail to examine the underlying societal shifts that are making people more susceptible to misinformation and conspiracy Trust in institutions is falling because of major political and economic upheaval Most notably ever widening income inequality the rise of automation is making fearful for their jobs Global migration trends are making people concerned that their communities will change the effects of climate change are becoming more pronounced around the world Putting homes and investments at risk as I said earlier when people are fearful and they increasingly are over simplified narratives conspiratorial explanations and messages that demonize others have become far more effective So finally, I know this is the web conference But we also need to acknowledge that the web is part of a wider information ecosystem We like to lay all the blame at the feet of the tech companies But the mass media and elected officials too often play an equal role in amplifying rumors and conspiracies And those of us who use these platforms also play a role in adding to the pollution when we choose to share misleading or divisive content This is my trumpet of amplification It's based on work by the likes of Whitney Phillips Joan Donovan Ben Decker, René de Resta But was designed as a simple diagram that newsrooms would take notice of it illustrates how so many of these rumors start in dark corners of the web for example 4chan, 8chan discord Then moves through the ecosystem via closed messaging apps like Twitter DM groups WhatsApp groups telegram Then moves on to conspiracy networks on platforms like Reddit YouTube and gab from there It gets pushed into Instagram Facebook and Twitter and too often it then gets picked up by the mainstream media Not that because they don't look backwards They think it's just appeared there and as we know sometimes when it appears in the professional media It then gets picked up by certain politicians and it then gets picked up by the media again So understanding this wider network and understanding how all of this connected is one of the things that we're not doing at the moment And when we look to platforms to self-regulate they are thinking about solutions and interventions for just their platform We have to think about this in the wider sense So this technological and social change has happened so quickly, none of us can keep up let alone researchers policymakers and educators We are well and truly over our skis and I know we're all looking for an easy fix But there just isn't one for the past three years I like many of you have probably gone to over a hundred panels and convenings on this subject right around the world I know we've been trying to figure this out But from my experience you either leave an hour-long panel so depressed You want to drink something very very strong or you leave a two-day workshop with a ton of Google Docs and post-its that nobody wants to throw away But everybody goes back to their day jobs and nothing gets implemented Any solution will need to be implemented at a massive scale Internet scale the major tech companies operate at that level but can and should they fix this themselves They're certainly trying but most of us including the platforms don't want global corporations to be the guardians of truth and fairness online Similarly the platforms are working to improve their own platforms, not the wider ecosystem. What is their incentive to do that? Facebook's third-party fact-checking project is admirable But Facebook is paying fact-checkers around the world to essentially build the biggest database of online misinformation But one that only they can access Shouldn't YouTube Pinterest Reddit and other researchers have the same access and at the moment the tech companies are marking their own Homework they tell us that interventions are working, but because they write their own transparency reports It's impossible to verify what is actually happening Often major changes to the platforms come only after a journalist uncovers bias or content that breaks community guidelines The tech companies have a very important part to play, but they cannot control the process And so what about the government those who call for regulation of the platforms seem to see it as the best hope of cleaning up the Information ecosystem, but lawmakers are finding it tough to keep up with the rapid changes in online and digital technology More important they have little access to data about what is on these platforms Besides which government would we trust to do this? We need global not national responses So the missing link is us I believe the people who use these technologies every day When conversations turn to the public it tends to just be about media literacy and too often a paternalistic framing of a public where people just need to be taught how to be smarter consumers of information But is there a way that we can rebuild an infrastructure that supports quality information? It feels to me that we're like one huge family trying to solve the world's biggest jigsaw puzzle We've just about found the four corner pieces and we're slowly building out the four edges as we define these challenges The jigsaw puzzle is going to take us at least 30 years If not more if not ever and our rush for quick fixes is actually slowing us down And the truth is that we should stop talking about solutions This is about building ongoing resiliency for a new reality Because we simply don't have the structures in place to work effectively for a problem of this size and complexity We would normally have a global body and my dream would be Would be on that that my dream would be that it would effectively include the public in the process But if we had a UN agency for disinformation, I don't think that would happen I believe what we need is an independent body that can identify design and facilitate the building of potential solutions in Cooperation with the brightest minds in technology policy and journalism All of this should be informed by insights and experiences collected from the global public Because if we want an information commons based on verifiability One that is held accountable and one that is truly representative of the global society Everyone has a part to play in solving this jigsaw puzzle. So what could we actually do? If we want to bring in the public, of course Wikipedia has shown us what can be possible It's not perfect, but it's demonstrated that with the right structures a global outlook and lots and lots of transparency You can build something that will earn most people's trust And we have to tap into the collective wisdom and experience of all users Especially people of color women and groups that are underrepresented They are the experts in disinformation and hate because they have been the targets of these campaigns online for a very long time But when they've raised flags over the years, they've too often been ignored. This has to change So what if we came up with a dramatically new deliberative process that involves a global community of concerned citizens Ready to share information and participate to improve our information commons There are four ways that I'd love to see the public involved One providing personal insight to have a way for people to share their experiences of particular issues not just opinions Secondly to donate data and share tips I first heard David Lazar talk about this in back in February 2017 And he was arguing that we're never going to get data from the platforms And he was saying how do we get people to donate their social data? That's their data. Sorry their social data to science Thirdly to join a micro volunteerism effort. This is personally my favorite thing in the world I love that volunteers do all of these tiny tasks like mapping our solar systems Tagging satellite imagery tagging artifacts in museum storage making new discoveries out in the wild and Every day the work that gets done on Wikipedia is eye-watering to me and finally taking part in experiments Nathan Matthias and his organization civil servant have inspired so many wonderful conversations about Running experiments on these platforms and he has worked on Wikipedia and Reddit. So let's take each one of these This image is about a massive consultation the UN did in the lead-up to the sustainable development goals It was an ambitious attempt to get feedback from the global public both offline and on There are some very interesting models that exist out there and many lessons to learn But doing this well is very difficult back in 2007. I wrote a report for the BBC about user-generated content It was all about who were these people who contacted the BBC via email or text. Why did they do it? One of the biggest findings was if you ask people, what do you think? Most people don't believe they know enough to contribute But if you ask them what they know you get incredibly useful responses People are experts of their own experiences But we have utterly failed to tap into that knowledge and expertise when it comes to these issues globally So many of the problems have been caused because these tech companies were founded here on the west coast of the US by really Well-meaning people but most of them had led very sheltered and pretty privileged lives The technology they have built is now used by billions of people around the world living in countries with complex ethnic and religious constructs, deep socio-economic challenges, histories of war, conflicts and colonialism Countries that almost none of these engineers have visited What if we created a truly global platform where the lived experiences and concerns of people could be reflected back Where cultural and political context and complexity could be explained in real time Where the abuse of images to fan the flames of hatred towards the Rohingya in Myanmar would not have come as a surprise When the violence related what's that messages could have been explained and potentially predicted through the lens of religious tension in India over decades When we take this historical landscape of religious tension and layer on a population when many people Via phones that they're only now experiencing digital technology for the first time It's much less surprising that we're seeing the spread of rumours fueling real-world violence I'm also obsessed with understanding which aspects of technology people are really concerned about Is it vaccine information? Is it hate speech? Is it news credibility? Is it deep fakes? Is it algorithmic bias? Is it addiction? I think there's a mismatch between what we're studying and building solutions around and that which most concerns the global public So this is a screenshot of the Facebook political ad collector that pro-publica built It's no longer working because Facebook changed the API But it was an attempt to get people to install a plug-ins They could collect data about the ads people were seeing on their feed Lots of lessons learned the underscore that we need to build a centralized open repository of Anonymized data with privacy and ethical standards at the core right now Everyone has a personal experience on these platforms and that makes it impossible for us to know what information people are really seeing It's impossible for us to run these audits Could we build an ambitious crowd-powered platform that would allow users to provide insights donate their data to science Provide tips about what people are seeing around the world to act as an early warning system Wait in on difficult moderation decisions and give feedback about changes that would affect the information users see Right now. I'm a fellow at Ted working alongside Alexios Monsalis who some of you may know He helped to found the international fact-checking network The work that we're doing is all about bringing the public into these conversations and to make them partners in the Interventions being developed. These are very early days So we don't yet have a community to draw on but we wanted to explore what it would look like if we collected experiences from the public So we use the platform lucid to get people in 12 countries to send screenshots from Facebook YouTube Instagram and Google We asked them to search using the search string. Should I vaccinate my child? We expected to get rude pictures. I'm not gonna lie, but we actually got amazingly rich data Here you can see examples of what you see if you search vaccines on Instagram These are just four of the countries that we looked at but as you can see hashtag vaccines killed vaccines uncovered There was a lot of pretty bad content on Instagram around the world and Secondly on Facebook, you can see here We asked people to search specifically so we could compare should I vaccinate my child and here's an example of a page I wanted to highlight this because it goes to show the power of emotional and Individual experience people here are sharing their personal stories of their children Facts don't work when you have images of children who appear to have had terrible reactions to vaccinations It's such a powerful case study for us to learn how to counter personal emotional narratives effectively And as you see they're talking about peer review when you spend time in these communities They share images of binders and binders of evidence. This isn't about people needing more information It's about understanding what's driving these communities And this is another interesting example The same article was getting featured in a top snippet on Google search But different paragraphs were being chosen one is paragraph 11 and one is paragraph 12 One is a solid description of why to vaccinate but the next one supports vaccine hesitancy and explains the times when it Doesn't make sense for a baby to be vaccinated fine with context But a snippet doesn't have the wider context and in the era as we move into voice activated technology You only have one answer when you ask Alexa. This is I'm particularly concerned about what we can do around snippets And we also ask people with no prompting What would you search for if a friend came to you and said should I vaccinate my child? And as you can see the number of different search strings people use is pretty eye-watering And this is just in English some of them and these are from some of the other different languages After we showed these the other day Facebook Pinterest and Google all asked for these search strings Which kind of flawed me for a second They made me realize that these companies have oceans of data But they're not set up to capture and understand the ways people actually search for these types of topics And so finding a way that we could simply buy build a tool for people to be prompted to ask what they would search for Would give us this incredible data data set of search strings So just imagine what we would learn if we built out this global network of concerned citizens who wanted to donate their Social data to science we would finally be able to examine how different algorithms impact what information we see in our feeds One of the major trends we're seeing is a chilling effect on speech There is a move away from the Facebook news feed into encrypted spaces whether that's Facebook messenger What's up or telegram? This is already happening. It's why Zuckerberg is talking about this pivot to privacy We've seen it in places like Russia and Indonesia and Brazil and Singapore as a result of the regulation being passed in these spaces I'm a firm advocate for the need for encrypted spaces But they do pose real challenges when it comes to the spread of false and misleading information The only way to understand what's in these groups is to build a community of concerned citizens Who want to act as an early warning mechanism to protect their communities? They can work with journalists researchers and fact-checkers to flag potentially false content so it can be investigated These people then can act as central nodes of influence in their own offline communities Teaching others and spreading quality information because these platforms are the opposite of the open web There is no broadcast functionality. It's mostly one-to-one communication or small group communications There's no way for quality media to share quality information in spaces like WhatsApp It relies on people to do the spreading This is why we have to properly engage the community make them part of the solution This isn't passive media literacy classes involving PowerPoint slides and an interactive game at the end This is going to be an ongoing attempt to engage the public in being part of the solution Another example is the project I mentioned that we ran in Brazil called comprova We had this central WhatsApp number and received over 250,000 messages That means we have the world's biggest data set of misinformation on WhatsApp But now there are issues about what we need to do to create frameworks around it What are the ethics of me sharing this type of data with other researchers? Can I share the data? It's come from encrypted spaces We have the ability to really learn from these type of type of data sets We're not going to get the data from the platforms But how do we work with concerned citizens and build ethical frameworks to enable us to do the work that we need to on that data? A database would provide so many things It would help act as a check when platforms say they've rolled out changes when we'd be able to do some community auditing It would act as an independent check when people say they flagged content But nothing has happened to it the platforms haven't taken a decision and We know almost nothing about how different types of disinformation and hate impact attitudes and behaviors over time We haven't even started doing the longitudinal analysis that we need to do and almost all of the research on these issues has been done in the US Though the problem is global So micro volunteers and this is galaxy zoo. I don't know if you've ever been on it It's a crowdsourced astronomy project It's a platform that allows people to help with the classification of galaxies There are many many many examples out here, but what would it look like in our space of information quality? Could we get people to classify different types of speech so that we could understand how people think about different examples around the world? And what they think should stay up or be taken down Could we train people to verify different types of content? They're seeing in different spaces online so we could start to understand patterns of content and campaigns that are being used around the world The process of classifying would also be the world's largest information literacy project as the process of verifying would be a way of teaching these skills And could we get them to investigate news sources around the world who funds them? Is there a corrections policy on the site? Do they label news versus opinion if this was designed properly? Not only would we be able to scale up some of the credibility ratings projects that exist right now But are struggling with claims of bias themselves. It would be a dynamic interactive news literacy project And finally getting people to take part in experiments I hope many of you know the work of Nathan Matias and civil servant They do amazing work with communities including Wikipedia to test out interventions in the wild So much of what we know about the ways in which people interact with information has been carried out by political and social Psychologists who've been running experiments mostly with American undergraduates in laboratories Could we design ways so that we can test and replicate these studies online in the wild to test whether interventions like flags Nutritional labels or more nudges eg. Don't share until you've waited two minutes might work So I'm a big proponent of throwing spaghetti at the wall and seeing what sticks rather than trying to design the perfect solution Let's try some things out for size. Let's prototype. Let's test and let's start again Finally we have to connect the dots no one sector and certainly no one startup nonprofit or government is going to solve this Wearing my first draft hat we pioneered the cross-check model where newsrooms work Collaborately collaboratively to investigate information disorder with newsrooms understanding that when it comes to these issues It doesn't make sense to be competitive So there are very smart people around the world working on these issues and many of you are in the room today from academia Civil society activist groups and newsrooms Unfortunately for the most part each effort is in its own silo and many are woefully underfunded There are also hundreds of people working on these challenges at the tech companies But their efforts can also feel disjointed as they're designing different responses to the same problems The project I'm now working on with Alexios Civic is designed to connect and amplify such efforts It's one attempt to find ways to coordinate the lessons from these different organizations and to connect them to the many Different people around the world who care deeply about building an infrastructure for quality information But this isn't about more email listservs or slack communities We need to physically bring people together in one location for days or weeks at a time So these problems can be tackled from different perspectives Ted has a residency program and from June 1st is going to be a residency program for Civic There are 22 desks and plenty of meeting room space We'll be running sprints this summer each Tuesday in June We'll kick off a workshop and then we hope people will spend the summer working collaboratively Some might be able to stay for just the day. Some might be for two weeks. Some might be for three months We want to identify problem areas We want different people from different perspectives to come and work side by side for a distinct period of time a time That will end with a prototype It might be a piece of code a policy framework an intervention to build friction into sharing or an educational resource Can we design a system where people safely flag content that they deem as potentially suspicious or downright dangerous? We need to work together to figure out the threats of gaming, but what might be possible? What does an open system for crowdsourced verification look like? What would an aggregated score for certain trustworthy sources of event information look like? There are so many questions to answer and prototypes to work on every week We'll open up the space for lunch so anybody who happens to be in New York can come and hear the residents explain What they've been working on so the wider community can provide mentorship guidance and feedback And what is crucial is that we evaluate at the end of this Which projects are a point which need more funding support from a platform or other organization that could turn a prototype into something That could really make a difference So if you're interested or you have staff in your organization or department that you think you could spare or if you have postdocs Who are looking for something to do this summer? Please go to civic central org and express your interest in part of this because we really have to do something about this It's been three years And I think it's really difficult for any of us to look at look in the mirror and say that we've really made any Substantial changes on this challenge these challenges of information disorder Because we need a coordinated ambitious response one that matches the complexity and the scale of the problem Together, let's reclaim our information commons by recognizing the complexity of the problem and working collectively on building long-term Resiliency and finding ways to bring the public into these processes. Thank you So I don't know if anybody has any questions. I think there are two microphones if you do Thank you very much for the talk. I have a question Your argument is the community should Collaborate and identify rumors and squash them down Do we want to allocate some mass of probability for their possibility the community is wrong? And I want to take you back. I mean fake news are not new in history, right? I want to take you back to early 17th century when an Italian guy was pretty ridiculous fake news And he was tried and the news were found to be untrue And he was forbidden to spread it anymore. His name is Galileo He was tried by acquisition for spreading use of heliocentrism, which we all believe to be true today What is the community is wrong today? So I think part of the issue here is that we like to obsess with what's true or false and what we're talking about here Is context and so my fear is if we have certain people who we appoint as fact checkers Or we say that certain people are these gatekeepers then we're actually losing the credibility of Portions of the public who are struggling with this the idea of bringing in more voices is not to say that you can tick this and say it's yes But in the model of Wikipedia what right now can we use in terms of transparency and evidence around something to provide that context? So yes, it's not saying that it's the answer But I don't think we have the answer right now and I think the absence of the public is just as problematic But do you think it should be squashed once once and for all based on the current knowledge available? Or should we allow for the possibility this knowledge will be reversed in the future? Yeah, I mean I think this idea of when people sometimes are critical of fact checkers They say well the evidence shows that fact checkers don't change people's minds Well actually for me the reason that fact checkers exist is so that we can have a public record of what was actually said And so I think in this context it's not saying that anything should be squashed You know those people who work in this space that we have this thing called zombie rumors Zombie rumors that just won't die and so it's not a case of saying if enough people do it We're gonna squash it it's about having more people in this conversation And it might be simply a case of you know as Facebook makes a decision. It's not about removal. It's about deranking So I think it's this problem is so complex and I think at the moment We're using old older versions of gatekeeping as a way of thinking about information And I think what does it look like if we bring more people in to actually provide that complexity? But yes, there are so many challenges with this it is tempting to go and sit under the duvet, but thank you I think we have to try something Hey, thanks for the cool talk was very interesting. I do appreciate that So I'm thinking that the problem of fake news has always existed It's just that with the social media and the internet it gets amplified and our weaknesses get amplified So I'm thinking that the pictures that you showed like of this woman walking by the victim or the immigrant attacking the nurse Whatever it was Whether true or not this has always existed and the problem is that for every such picture I can find 10 pictures where a Non immigrant attacks an immigrant right so isn't the problem rather That we should educate people to not take every single picture even if it is true as evidence of their own theory I think there is the problem in this interpretation not in the sharing of the picture. What do you think? Yeah, and that's why I keep talking about resiliency is that we haven't done a good enough job of explaining to people that when you're Standing in line for coffee and you see an image if we don't teach emotional skepticism We're being used as weapons against ourselves So the resiliency is not we don't need more media literacy to say is this a headline is this in our bed? How do you do a reverse image search? It's about saying if you see content that makes you have an emotional reaction It either makes you angry or fearful Stop and think about why are you resharing it? So I think the resiliency piece is what we have to do and thinking globally where we have many communities Who've never had an email address and have gone straight to phones, but they're incredibly trusting because they've got these fancy phones What something that's on a fancy phone? Why would that not be true? So we fail to really think about what media literacy looks like in this context and we've tended We've fallen back into older media literacy curriculum and we haven't said what do we do now to build resiliency for all those things which is The only solution here is about the public and educating the public and part of the reason of trying to bring them in is My fear is that the conversations right now about the public are very paternalistic, but making people part of this I'm hoping we'll make them recognize the complexity by doing something as opposed to just passively receiving information But I couldn't agree more and I would say when you know when people started worrying about this Some people said what about building resiliency in the public and every said that will take too long What we need is an algorithmic tweak and I think three years on every's like oh that algorithmic tweak isn't gonna happen And really we have to recognize we need 30 years of resiliency training because it's not just for children You probably saw this research the people who are most likely to share misinformation are older white men So thank you. Yeah There's none of those people here All white men to the mics We can take a couple of more questions There are two mics in the back. Hi there. I love the idea of Galaxy zoo as an astronomer. I love the idea of Galaxy zoo for news One one problem here that then we may have One obstacle to overcome I suppose is that you know Galaxy zoo was this the Quintessential gamification of science. It was so much fun and there was no context outside what people were doing of politics Yeah, so how do we overcome this this huge obstacle? And that's partly why for the beginning trying to get people to do things like classify speech I mean to get to talk about regulation for a second what we're seeing is separate governments thinking about regulation But this is actually a really big global problem And I think what we haven't understood is is it possible to have one standard for speech or actually are there different parts of the world you say No, I want to dial and you know, for example, we know this in Germany They don't allow any Nazi related content on Twitter. And so one of the pieces I'm interested in terms of classifying speech is getting people to just kind of give a sense a kind of a gut reaction of How they respond to certain things as a data set So I think one of the things I think about micro volunteerism is just helping us get a whole load of different types of data And then making a decision about what we could safely do or what would work with the public in a kind of a more Galaxy Zoo way But I do think it's a powerful way of just getting a sense from people about what decisions they they would like to see Absolutely brilliant. And then of course you carry on and use that as a training set for something That's more scalable. Yeah, because I mean all the engineers in the room The problem we have with all the interventions right now is very well-meaning nonprofits doing things like tagging the credibility of a new site But they've got data sets of you know, 200 and engineers need much bigger data set So that's the other way that if we can scale this could we do something amazing? Hi Claire, thank you for the keynote. So I Have a question. So we have we developed a platform where Users can donate their ads and we are trying to bring more transparency to understand Who are these advertisers and what are the message in there in their ads? And my question for you was that we are trying to to build a bigger user base and I was wondering whether you have any advice on how to incentivize people to install the tool so we can get a better view and Which country are you doing this in? So we I mean we are I'm from France, but we also did it in Brazil. We collected some users and it's A worldwide development is not specific to a country. Oh great Well, also we should talk afterwards because I want to connect you with some people But the real question here is how do we amplify this? So for example in Brazil when we had the cross-check project where she had 24 of the largest Brazilian newsrooms So the reason that we got so many messages from WhatsApp is because you had 24 very large newsrooms pushing out the same telephone number So I think there's a question here is like how can we adopt news organizations? But also think about other communities. So if we're thinking about 2020 in the US, how can we reach different communities? And that's about church groups. That's about other activist groups That's about thinking about where how can we reach different communities who don't normally install plugins because Pro Publica? We love Pro Publica, but Pro Publica's audience is particularly to the left So all the ads they were getting in were from people who were particular So then they started paying for Facebook ads so they could actually reach more conservative Readers and that changed things so I think thinking strategically about how where are those amplification? Organizations and some of them are going to be newsrooms But not only newsrooms because there are big sways of the population who no longer trust the mainstream media So we think we have to think about different things, but now I would love to talk to you after about some of these things Feels like the Solutions slash Directions that you've been discussing kind of operate on this what I'm calling a what kind of level where you have a lot of maybe data points of some form where they're tagged as either You know true in some sense or not true But it seems to me that you'd have a lot of people who could just kind of systematically deny that in the same way that they do And they wouldn't be unreasonable And so I'm kind of wondering what would a why Approach look like where your data kind of had structure that showed not just here What is what we think is true here is how all of the evidence and other things that we know point to why we think that's true No, you're spot-on. It's you know I talked about the weaponization of context right now We're obsessed with what's true or false or this fuzzy middle of misleading and what we're not providing is the wider context of you Know once you know that that tweet was from a Russian disinformation bots It changes that so I think so much of what we don't know right now is the why and we haven't got mechanisms because Structure and databases are around the things that we can you know There's no subjectivity and unfortunately there's a lot of subjectivity here So the structure you're talking about is how can we build structure around subjectivity and context and I'd say that's that's where we need To go and one of the reasons for doing these talks and throwing these examples up is I think too often when I go and talk to Students, they're still obsessed with the true or false And I think what we need to do is these data sets of the middle to say that's what we really need to be focusing on Because we're getting much better at the automated fact-checking of the true and false what we can't deal with is that with the why but I haven't thought about in Those terms, but I will show you use that from now on because that's very smart. Thank you. Cheers Thanks for the great talk. It's a topic close to my heart. I I'm part of a bunch of whatsapp groups where I get these messages and I'm the fact checker of most of those groups So I know I keep bashing them the phenomenon I see is In those groups, I see fewer of those messages trickling to me like the friends and family They have started selecting what messages they forward in the groups because they know This is suspicious. So I might get bashed Is this a widespread phenomenon like, you know the fact-checking resistance? So you don't send to groups where you think there are fact-checkers sitting You just you know people don't like fact-checking. I think this is a widespread phenomenon. What do you think? Yeah I mean it's a human phenomenon people don't like to be told that they're wrong So you're absolutely right, which is you know, we all struggle with confirmation bias So for example, you know this this example gets used a lot But you know if you're scrolling through and you see a piece about Obama sending illegal immigrants back from the left You want to scroll past that because you don't want to read it because it doesn't support your worldview So it's not a right or you know, it's it's a human thing But to your point we have seen that too Which is there are more people now getting active in whatsapp groups with their families and they're seeing their families mute them Or kick them out so in a place like Brazil that we did a lot of work It's really sad to see the same kind of things happening as we saw in the US Which is families being torn apart because people don't want information that challenges them So that's the biggest problem we have in all of this. Thank you Hi, my question is that So for people hear about rumors and hearsay that just something kind of normal in our daily lives But the same that one does those kind of questionable information coming into a web of people start to kind of question Why so if that's a case that the people kind of assign a more trust to something be written Then something be heard and then when those information come to the web and the people get maybe confused What do you think about that because to some extent if you believe that in our daily life with rumors are kind of normal Maybe those questionable information start to showing on the web is a kind of a truthful representation of our daily life Yeah, I mean, I think this format question about text versus video versus audio is really interesting Which is trying to understand how people think about credibility and one of the challenges in a WhatsApp group is there are very few Heuristics those mental shortcuts, so you don't even see who it's forwarded from you don't see where it's come from And so the only heuristic you have is it's from your best friend or from your mom And so you're much more trusting of that so to your point We have so much to learn about credibility when it when it's about different types of rumors We've focused so much on news credibility like do people trust the New York Times or the Washington Post We've done a lot less on how credibility works in the online space, particularly text image or video 10 seconds for the last question Okay, I'll try really quick so to me this fact-checking in the problem that you're trying to solve feels a little bit like helping victims after the terror attack because That that seems like it's not possible to do it just to re-handle the results What do you think about actually solving the problem by? Go into the source and like if we know that there is a source that's disseminating this then there has to be active action taken not maybe on a information level but on the Military level so like if we knew that When we knew that Al Qaeda was doing things that people went up to Al Qaeda if you know that there is an entity that is Responsible for all of this. Why isn't it just a solution to? Take measures. Yeah in other fields. Yeah, no I mean one of the things that we struggle with is that the way that Russia used Facebook was the way that Facebook was designed to Be used so it's very difficult often to know who's behind this Facebook is getting much better at this And yesterday you might have seen they took down a huge amount of networks from Israel that were targeting African countries And when you read that you just think this is the tip of the iceberg because it's so easy for anybody to pretend to be Somebody else and to influence and so to your point. Yes. Yes. Yes The challenge is again because the data is locked up in Facebook We have to assume that Facebook are looking and the rest of us can't do anything about it But yes, we might and rather than thinking about regulating or taking down content We should be thinking about regulating and taking down behaviors But it's much harder to get access to data that shows us those behaviors Let's thank Claire So here is where we need your help all of you who are sitting in the back Please move forward Everybody who's sitting on the two sides. Please come to the center if you are one of the organizers


Patriot Prime Reviews is a channel for adult collectors and may not be suitable for children under 13 years of age. Viewer discretion is advised. Hey, what's going on guys? Patriot Prime here once again with another toyhacks.com sponsored review. But before I get started, I need you guys to do me a favor. If you're watching this video right here and are not a subscriber of Patriot Prime Reviews, please consider hitting that subscribe button right now. It won't cost you a thing, but will help me and my channel out tremendously. And help me reach my goal of 10,000 subscribers by TF-Con in October. Now, on to the decals. And the set we're looking at in this video is for Generation Select's Sandstorm. This decal set includes four decal sheets, three on foil backing, and one on clear. So now, without further ado, let's take a look at Sandstorm both before and after toyhacks decals. And welcome to Patriot Prime Reviews. In robot mode, straight out of the packaging, Sandstorm looks alright. He's got great sculpted details, but only a bare minimum of paint applications. Taking a closer look at the figure, you've got some white here inside the pylons with some red triangles there.  You've got some black here. And black down here on the little thrusters and the feet, along with the thrusters there for vehicle mode. There is some white and red on the knees. And really, that's about it. My biggest complaint with this figure in my original review was the bot mode's details pale in comparison with the vehicle mode. But hopefully, toyhacks will fix that. So there's one final look at Sandstorm in robot mode before and after. Sandstorm's robot mode gets quite the upgrade thanks to toyhacks. Taking a closer look at the figure, you get new decals right up here in the pylons.  Here on the chest, he gets new chrome decals for the intakes. Black and purple decals here for the waist. Black decals on the shoulders. Black and red decals for the wrists. Moving down to the legs, he gets new purple decals for the knees. And black decals there for the outline. Down here on the feet, he gets new black and purple decals. And G2 Decepticon logos there for the inside of the wings. And of course, you have options where you can go old school Decepticon, but I decided to go G2. He also has intakes right there inside the pylons. So I am really impressed with his robot mode now. So there you go guys, Sandstorm in robot mode after. In vehicle mode, straight out of the packaging, Sandstorm looks great. I love the details on his jet mode. Of course, that could be just me because I do have a history with that particular paint scheme. Now taking a closer look at the jet, he's got the great flying tagger details right there on the nose cone. Desert camo all the way around the jet, aside for this area right here. You got the black thrusters. You've got the tail fins painted a tan color to kind of match the rest of the jet. And let's see, black back here, which is the robot's feet. And that's pretty much it, but it does look great. And I really wished that we got that type of paint scheme in robot mode. So there you go guys, one final look at Sandstorm in jet mode before and after. Sandstorm gets quite a few new decals to help enhance his jet mode. The first ones you're probably noticing are the new purple decals there on the tail fins up top and along the side. He also gets new generation two Decepticon logos on both wings, and those are optional. You do have the regular Decepticon logo and tail fin logos there that say DC, but I like the looks of the purple. He also gets new decals right here on the back of the wings.  That is really cool. That blends perfectly.  And I'm a big fan of these because if you watch my channel, you know I love the show V and that looks just like the logo. There's also new black decals right there. A new black decal here on the front. New decals here along the nose cone. You have some options. I have the one that says Bot Eater with a robot shark eating a missile. A little red logo right there here on the other side. You have Sandstorm's kills. I love how that looks. That looks sweet. The options, of course, are just some plain logos. You got a red triangle there and rock the Kazba, but I loved the robot shark. I thought that looks sweet. He also gets new decals that extend the mouth of the flying tiger all the way under the jet and gives him weapons or machine guns underneath. And I think that looks awesome. All in all, Sandstorm's jet mode blew me away before, but now it rocks. So there you go guys, Sandstorm in jet mode after. Toy Hacks once again knocks it out of the park with another generation selects decal set. I really like what they did for Sandstorm's robot mode. As I said before, I thought his robot mode was way too plain. Now I originally had my doubts with the purple decals, but they work. And I guess since he's G2 and G2 had those garish collars, the purple really works for Sandstorm. Jet mode, I love what they did with jet mode, especially with the extension of the flying tiger mouth on the nose code. Now guys, if you enjoyed the video, don't forget to like, don't forget to subscribe, and don't forget to click that bell icon to get notified when I upload new reviews. Once again, this is Patriot Prime, signing out. Ho-A!


Hello Aces, welcome back to module two, lesson 6.1. Today we're going to be talking about how do you craft a winning and profitable menu for you. This is a really, really important lesson because we create a menu, not just that your customers would love, but also allows you to be super profitable. And that's exactly what we're diving into today. In this first half of the lesson, we're going to figure out exactly what your customers are wanting, job to be done. Okay. If you haven't already done so in the last lesson, we talked about what is the job to be done. Make sure that you complete this before moving on to this specific lesson. Otherwise there's no context on what you're doing. Your product offering specifically beside each of your job to be done that you have completed in the last lesson, put down your proposed product offering that fits and that actually accomplishes and solves the problem to be done. What is the item that you would like to offer for each job and function as a response or as a solution to that? Okay. For example, McDonald's, they offered milkshake to fulfill the job to be done, which is something to fill me up during commute. For you, it would be something a little bit different. So in the worksheet down below, download it and start working on it together with this lesson. Complete as much of the product offering as there are for jobs to be done for your ideal customers, helps cover all the bases of the items that you would like to offer. Remember, we don't sell what we want to serve. We sell what our customers are wanting, what they're needing, and that's going to be able to increase the number of sales you get. As an example, if we're offering ice cream, what job does it satisfy for a family of four who's visiting your shop as a treat before they head to the beach versus an Asian couple that's coming over for date night? Because the job is different, the offerings would be different as well. For the first half, which is the family of four, we would be offering chocolate flavored ice cream cone with lots of sprinkles because kids love sprinkles, kids love chocolate, and they can't have any caffeine items and they don't, their taste palates are not as sophisticated, which is why every kid that comes in through our doors or offer or are ordering chocolate ice cream. So if you don't offer chocolate as an item, then that's a big mishap on your part versus an Asian couple whose taste buds are a little bit more sophisticated. They want to have a better experience. They want to have something that's novel. We would be offering them an Instagram friendly smoking matcha flavored ice cream and that's precisely the reason why we're so popular is because we understand who is it that we're serving. Are we serving to families or are we serving to couples? We serve to the millennial couples and that's the reason why it picked up like wildfire. So now it's time to build your winning matrix and it has four different components. We figured out the job to be done. We figured out the offering that you would like to offer them. Now let's figure out the math part. Is it profitable? Is it not profitable? The strategy behind the items that you're serving. Let's figure it out guys. In this winning matrix, there are four different types. There are the stars, the cash cows, the hidden stars, and the skinny cows. What are stars? Stars are highly profitable and highly popular items within your menu. These items are emphasized and should be focused on because you want more and more people to buy it. Aim to have three of these types of items on your menu. Cash cows are low in profitability but they're high in popularity. These are popular items that customers come in all the time again and again for. So for example at McDonald's, one of their cash cows are the cheeseburgers. As much as it's not sexy, it is not fun, but it is a staple item that a lot of people go to McDonald's and purchase. It is high in popularity, but the profitability on that is not that high. Profitability usually is higher on more expensive items like for example their Angus beef burgers, right? That's going to be something that is higher in profitability, but they're not as popular. Next up is that an aim to have five of the cash cows in your menu.  You want to stay competitive. Hidden stars are the high in profitability but low in popularity. A lot of potential for these items when you are able to market it properly, when you can actually repackage these items to make it more popular, then they become more and more popular because you want to be able to have these items that are really high in profitability and change them into something that a lot of people order. You may want to be able to perhaps lower the price a little bit so then it becomes less profitable, so then that way it becomes more popular. But a lot of times what I see as an alternative to making the hidden stars a real solid star is to actually repackage and repurpose these items. So a simple dish of pasta, not much flair to it, might not be as popular as truffled meatball spaghetti. That itself already sounds super, super enticing and in turn becomes something that is going to be high in popularity and high in profitability as well. As I said, you may need to repackage these items, rename them, reintroduce them, placements of the menu. There's a lot of ways to turn these hidden stars into stars for you. Now, we talked about cash cows, now it's time to talk about the skinny cows. These guys are low in profitability and low in popularity as well. Cut these, we don't need the skinny cows. They're just basically exhausting our menu space, exhausting our kitchen space, and it just complicates things. A lot of times when you launch a menu, we want to avoid the skinny cows. However, we wouldn't know until we actually put it out in practice and to actually see the results. And once we see the results, we can always go back and change these. Now, a special star that I'm sharing with you is the shooting star guys. What are shooting stars? Shooting stars are decoys. These are super high valued items, which you're not really meant to sell much of, but these guys would make your stars and your cash cows much more attractive by actually presenting them at a higher price. So as an example with our ice cream, we have a $50 ice cream, a $50 ice cream with goat flakes, with caviar and everything on this. And it's like 50 bucks. We don't, it doesn't taste good, not necessarily. Is it, is it popular? Not at all. But people will order it just for the grand. Not a lot of people order it. And for us, we don't really care much for it. It just puts our stars, our cash cows in comparison, a six buck or $7 ice cream versus a $50 one. Well, it makes the $6 one much more affordable. It makes it much more attractive. And that's the reason why we have these shooting stars in our menu. It's just meant for us to actually as a decoy. Okay. Now it is your turn. Our goal is to craft a menu full of stars and cash cows. At the end of the day, it's going to come with a lot of time, reiteration and revision. Okay. Ongoing process with testing, adjusting and fine tuning for the stars and to grow your cash cows. Okay. It does not happen overnight. The reason why you did your customer avatar, your job to be done, and your product offering worksheets is for this moment guys, is for us to actually understand what is it that they're looking for? Who is it that we're serving? And by us doing all the ground work, now's our time to actually identify the offering, the what is it that we're serving them? And the lesson below, download the worksheet so that way you guys can start working on this. Besides each of the items that you create, you already created for the job to be done and your product offering, list them up, whether they are potential star, potential cash cow, or hidden star, or even a skinny cow. Just write it down. For example, something for your commuters to do and a snack for your commuter. And then we put it down, what are we offering? Milkshake. Milkshake is a cash cow or is it a star? I would say it's more of a cash cow over a star. And that's exactly how you want to be able to craft your menu. Your goal is to have at least 10 items on your menu, three of them are stars, five cash cows and two skinny cows. Okay. And zero hidden stars. The point of this exercise is so that that way you can plan for success. If you don't already plan it, if you don't put this into motion, if you're not aware of what you're creating and intentionally and strategically putting this in your menu, then you just putting stuff on the wall and hoping for things to stick. And oftentimes you'd be running maybe half a year in and not realizing that, Hey, you know what, my menu is not that profitable. I need to increase my prices.  And that itself creates even more headaches for you in the future if you don't start off on the right foot. You probably don't know where your proposed items fall within the matrix before your launch, which is completely okay. However, it is all about actually going out there and start doing things. Okay. You need to plan for success and keep innovating until you have that perfect matrix. And that's exactly what we did. We keep reiterating, reiterating, take things out, add things in just so that way we can make our menu super profitable. Keep this matrix sheet that you're working on and always come back and revise it.   Follow this as your blueprint as your guide for your launch. What you have learned in this lesson today is how do you craft a winning menu that your customers would want the job to be done and that is profitable so you can merge them together, create a profitable restaurant. Next up, we're going to be talking about the six menu design hacks for your menu. So make sure you guys keep watching. I'll see you guys in the next lesson.


Hi, my name's Katie and I'm going to be teaching you how I teach young children about the life cycle using millworms that go through metamorphosis. So, together in a group with my children, I will talk to them about how they grow and how they used to be babies and how they get bigger and that's kind of how we'll start the process and then we'll move on to introducing the insects to them. In a small group, we'll sit together and we'll discuss what a larva is.  I will explain to the children that the millworms do not have teeth and they cannot hurt them and if they'd like to touch them very gently, they may. If they don't want to touch them, they don't have to. I'll let the children observe the worms for a little while and then after they've collected some thoughts, I'll come back and I'll ask them what they've noticed about them and we'll make a chart labeled Observations of Millworms. Then we'll move on and we'll talk about the pupa. The pupa might be more familiar with them as a cocoon. This is what the millworms go into as they transform into a beetle. We'll discuss that with the children for a little while and I'll allow them to make their observations and then we'll also chart what they have learned. Then we'll move on to the beetle. I'll tell the children that the beetle used to be a worm and we'll talk about how it transformed into a beetle. I'll let them make some observations and then we will have a discussion together and we'll chart what they've noticed. Then we'll come back and we'll revisit our chart and I'll also show them this flyer about how in the beginning stages we have an egg and that egg then turns into the larva which is the millworm which then turns into the pupa pupa or cocoon and then becomes the adult which is the beetle. So that's how I teach young children about the life cycle.