Welcome to Free Thoughts. I'm Aaron Powell and I'm Trevor Burrs and joining us is Will Duffield He's a researcher with Kato's first amendment project. Welcome back to Free Thoughts Will. Good to be back What is Cambridge Analytica and what have they done with my data? So Cambridge Analytica is a political consulting firm and we aren't perfectly sure what they've done with your data Now to start This is a story full of unreliable narrators has some confusing twists and turns and a lot of those with access to privileged information in this story Have a whole host of incentives to misrepresent what they know The basic story of the Cambridge Analytica Facebook data scandal is that a researcher named Alexander Cove Logan, back in 2013, used Amazon's Mechanical Turk Service to hire individuals with Facebook accounts to take a personality test which Under Facebook's rules at that time Also allowed him access to a basic set of information about these test takers friends This is one of these tests that you sign into the test online with your Facebook account to answer silly questions And then it posts the results to your Facebook which which Harry Potter Hogwarts house. Are you kind of thing? Yes, that sort of thing exactly This data was then Contrary to Facebook's terms of service sold by Alexander Cogan to Cambridge Analytica They may have used some of this data both information from the test takers and information about the test takers friends to create psychographic profiles used to Potentially more effectively target political advertisements. So this as you've described it sounds like much Less of a big deal in terms of scope than the headlines have said so we're talking about some people Five years ago who took a personality test So how do we get from that to the like hundreds of millions of Facebook users? data has been exposed a couple hundred thousand people took the test and each of them had a few hundred friends so Initially, it was expected that 23 million Facebook users might have had that simpler set of data the friends data Pulled and incorporated in this data set There were a number of other tests that Cogan had run around the same time Which is how we get the revised number of potentially 87 million accounts and affected what kind of data do they have Was in this bundle that they managed to gather both on so there's data from the people who took the test And then there's data about the friends of the people who took the test, but what's in that data? So that will include who your friends are Your age employment status Some location data this is What you've chosen to post about your location your home town or not high school not Data gleaned from cell phone records things you like would that be in the data certainly yes So, I mean the theory is you've like yeah the theory here is that you could I mean this would be Advertisers in general you could use that to figure out that if you liked some page for I don't know buck hunting enthusiasts They might want to advertise some some shotgun shells Actually, you don't hunt deer with shotgun you will advertise some rifle re equipment But that's what it seems pretty innocuous in Cambridge Analytica was a political consulting firm And they used it for that purpose and Ted Cruz had used it before Donald Trump did correct. Yes Though I believe Ted Cruz's contract was originally with SCL group Which is a larger parent firm which has done Much more government and even military work in the past There's another firm in the mix under this SCL umbrella called aggregate IQ Which was more heavily involved in? Britain's Brexit vote leave campaign though so I mean my my first reaction is See so what that was my first when I heard about this I said okay, so what? Is that a good general reaction to have or is it? Okay, but or is it something that will opening up a discussion that is gonna be with us for a long time I would say when it comes to this specific instance how This data scraped by Kogan might have been used in elections. It is a sort of so what response I Think a lot more of this has been going on as well that we don't know about but this poked above the surface we've latched on to it and Because they're a global firm. They were involved in elections both in the US and in Europe It's been easy for all sorts of people to see a niche story that applies to them within this broader concern now it does presage a sort of concerning state of affairs when it comes to Data in general how and how it's used However, a lot of that contingent concern is also reliant upon your theory of political communication You said this was What was done here was against Facebook's terms of service so you can so Facebook would give this data to a third party in this case The the guy running the quiz, but you weren't allowed to then pass that data on to yet another party yes, and When we talk about Giving data for the most part it was it's the app itself. That's collecting that data It's not so much Facebook Facebook provides the platform for it Sets the rules governing apps in general what they can pull But the users are then authorizing the specific application. They're using To gather this data so when the user went to take the quiz they had to click through something that said Do you want to share all of your data with this third party and they said sure yes including the friends data so Zuckerberg goes to Congress who seemed to be quite upset about this slipped out about this and What were you you I think watched all that or more most of it? Mercifully, yes Well, I saw some of the highlight reels going around but but what would be the if you're having set through the whole thing What would be your description of that? Bizarre event ignorant The Most prominent misconception displayed in those hearings seem to be that when Facebook Advertises on someone else's behalf the advertiser is in the minds of many Congress people Gaining a tranche of data from Facebook to use for its own advertising purposes That's not the case When you seek to post an advertisement on Facebook pay them to spread a message You choose an audience within Facebook Sort of people who you would like to reach and then they Using the data that they have on us Serve that advertisement to these populations So it's not that it's it's again It's somewhat new in the sense that we have more data about people than ever before But it's also just Running at I mean while we've always run ads We tried to avoid showing ads to people who don't care about them. That's pretty difficult It's just a more sophisticated version of saying like this television show is popular in the 18 to 34 demographic of men with incomes of certain amounts, so I'm going to Target and quotes that audience by advertising in front of that TV show yeah, I remember I was a kid and watching Saturday morning cartoons and and Always going to my parents and being like have you seen that commercial for you know this new play set? I know we haven't seen that come right. I was like what how could you have seen it? It's on it's on every every time It's always on and if your parents were watching along with you The money spent to serve an ad to them was rather wasted. Yeah, there's an inefficiency there It's it's like it's like why I the NFL For some reason decides that the only thing that should be advertised in NFL games is pick up trucks and enlisting in the Navy and Rectile dysfunction. Yes, and buzzer none of which are necessarily anything I would consider buying Yeah, so what are we now if this is the nature of the modern world and people seem to understand that? I mean I think people understand that a lot of data Is kept on them by a bunch of corporations I think they don't actually care that much which we could talk about they should And they use that data and so will politics so so Is there more to this in terms of like well, I mean going forward There'll just be more data and more specific data and it could be you know Where were you at Thursday at 6 o 1 p.m. Or whatever? And so does he get creepy at some point well I think there is certainly a sort of civic Republican concern that accompanies the use of targeted advertising and politics if your neighbor is receiving different Cola commercials than you no real reason to care about that if advertisements are you political advertisements are used not just to convince but to inform and Increasingly our electorate lives in different information spaces We can see a concern there But that also comes along with the usual individual choices of people to consume different sorts of partisan media So it isn't a new concern, but it might be magnified by this and what about Russia? That's the other one way the other one that manipulated I'm putting that in scare quotes Manipulated our particular political processes with via Facebook. What did Russia do? Russia ran fewer political advertisements in doing this in Attempting to interfere They weren't necessarily what you would think of as an electioneering communication They were mostly instead designed to Provoke partisan or tribal fears Things like Sharia law coming to a community near you or on the other side Discussing police brutality against African Americans So they they don't fall within what we would think of as Political communications, but they were definitely designed to heighten tensions that already existed within the American electorate And they often aped the style of some of the most inflammatory voices in our polity Trying to understand the privacy concerns Here the I mean because clearly the response to this story It's clear that a lot of people think we've already gone way across the creepy line And in Congress thinks that we've gone across the line that upset them in a way that the Snowden revelations didn't but so bracket the question of When Facebook five years ago was allowing apps to scrape this data which it sounds like they They've restricted that then that in 2014 so that's not that's not a thing So so the data is being gathered and aggregated by Facebook and it's being kept in Facebook servers And so this is why that you know the hip thing a couple of weeks ago was to go to the Facebook page Where you could download all your data and then dig through it and be Astonished at how much you being on Facebook six hours a day generates But but Facebook as we said is not they're not sharing that data on an individual basis in any way like with With outside firms. They're just saying like if they know that you happen to be a certain age They know that you happen to like certain things and so if someone says I want someone of a certain It I want people of a certain age and people who like the following things to see this ad Facebook will just marry them up But but is the concern then that the data, you know from like a personal privacy issue That it could get off of Facebook servers that Facebook could get hacked or are people concerned by the very nature of this aggregating That's a concern, but it does go to the aggregating and it involves Facebook and even the internet Less than we might expect The fact of the matter is there a host of Interactions which create data Which prior to the past decade or so was very difficult to capture and organize you're having purchased something at Target or Buying a certain insurance policy Couldn't be brought together into some kind of cohesive whole picture of your life That's changed now So a host of things we do on a daily basis which previously couldn't be very well tracked Now contribute to a mosaic of our lives and I think that is Uncomfortable for many people. Do you think that that discomfort? So like as a libertarian I am uncomfortable with the notion that that kind of data Exists in a silo somewhere even if Facebook's not giving it away Facebook is still Exists within the territories of states that would love to get their hands on that data often to Do bad things to people who are not necessarily fans of that state And that makes me uncomfortable because this is you know access to something that looks very much like incredibly pervasive surveillance But from the the private end is there I mean Facebook's just using this to give us targeted ads and I think everyone would admit that On the whole More targeted ads are better than less targeted ads just from the user experience Yeah, I'm tired of that cars for kids commercial if we could you know target something more Salient to my interests that would be an improvement. So What's the what's the privacy concern? Outside of just like I feel this is feel like this is creepy is there like you know call it from our libertarian perspective Is there like a genuine privacy concern here beyond that states might get this data? I would say there still is particularly when one's expectations Concerning how this collected data can be used to discipline them are at odds with the extent of that reality if you used to have an Ashley Madison account But didn't think that that was the sort of thing that a Co-worker could find out about and use to push you into taking on some assignment at work Well finding out that it can and they will is Discomfiting It alters the ways in which we might comfortably interact with the world around us How much do you think the uproar about both Cambridge Analytica and Russia in this specific Facebook in Russia is related to Relitigating the 2016 election. Oh very much so Now I don't think that Either the Cambridge Analytica scandal or Russian involvement here or even work that a QI did on the Brexit campaign Altered the outcomes of those elections. However, the ability for Effectively marginalized groups within our polity to use social media writ large Probably had a very real impact on the election so Cambridge Analytica and Russia are used as a proxy for that broader concern that these voices are Coming out of the woodwork and there doesn't seem to be a very good way to hush them up again But what's the problem with that? I mean, what's the problem with more voices coming out of the woodwork? Is it that they might be Alex Jones? I think that's what they might be in this case They voted for Trump. Yes, right But but these are I mean we've always had candidates have bought ads all the time And we don't have a problem with that and political parties buy ads all the time and government officials campaign all the time And you've had organizations like what does that guy's name David Ike? Yes The lizard. I mean he's been putting that stuff forever about that, right? Right. So I Guess I just I have a hard time feeling the weight of the concern that Lots of prior Marginalized voices are getting their say I mean I get that a lot of these Marginalized voices are crazy But that just seems to be more a problem of like why are so many Americans? susceptible to believing crazy stuff But I don't I don't know that they're necessarily like these voices are more dangerous than hearing your local politician Tell you to do something. Yes, and there's certainly no just reason for depriving these people of access to contemporary telecommunications technology I think that the the issue is and I've talked about it before on free thoughts and episodes that we talk about campaign finance in particular that If you don't understand why people disagree with you And and there's something we can get we'll get into bubbles and stuff like that too And but if it is a case that if you have no good story for why someone disagrees with you except for they they must be duped by something and I think there's some of it as it says that we Misunderstand the other side more than we used to to some extent And so therefore you need to have an explanation for why this thing happened like Donald Trump's election And we were really explaining other people's political beliefs and trying to come up with some sort of reason why they believe these things and That's what's scary to me That's what I realized when I saw this Cambridge Analytica thing that this is the new campaign finance This is the new citizens united discussion the worry about the corporations was always that they would have so much political Power to run ads that they would convince people to vote against those people's interests I'm putting that in scare quotes and for the corporate interests. And so we're afraid of you know Any of these terms you're like someone's corrupting our democracy or we have Cambridge Analytica corrupting our democracy or corporations Corrupting our democracy and that always makes me just think interesting because I'm okay corrupting So how is speaking to people corrupting a democracy and there's so many implicit premises in that? And I think that's what we're getting into with this Facebook stuff Yeah Where you draw the line as to what is endogenous versus exogenous to one's democracy? Tells you a lot about their theory of political communication That's a legitimacy of different forms of messaging or messengers. There's some sort of line from Influenced to brainwash right and it runs through manipulation in the middle And so you say okay You're you're allowed to influence the electorate and you can get out there and make your voice heard and but you can't brainwash them and Manipulation is probably too far too And so I think that they would probably put the people who are afraid of Cambridge Analytica future of this They'd probably put it at manipulation and they're probably worried that if the data gets good enough It could go to brainwashing where they have some sort of algorithm that says if you wash your car every Tuesday And you buy kumquats and delight to play squash We just have to show you six things Like a quala bear video and then like a video of Donald Trump and then this and then you will now vote like we'll crack Your brain like some sort of you know Combination to a safe and I think that you know me that's very sci-fi But that's somewhat what people are also worried about I think it is but especially in the political realm I feel as though those fears are presently overblown When it comes to Cambridge Analytica in particular, there's very little evidence that Any of this worked or even was perceived to work by certain campaigns that hired Cambridge Analytica you look to Cruz's use of them He was doing it because they were the Mercer's data analytics firm Robert Mercer funded Cambridge Analytica. He also funded a number of GOP campaigns this past election cycle and One way in which you could signal goodwill towards him and ask for his money was by hiring Cambridge Analytica When it comes to their actual services, I was Supposed lucky enough to receive a sales pitch from them about a year and a half ago I'd been working for a cannabis policy magazine in the UK that was looking to begin a legalization campaign and One of the firms we had in to discuss Potentially working on this campaign was Cambridge Analytica They played up what they could do a sort of slick monorail salesman-esque pitch But at the end of the day it relied on a great many gimmicks Posting a bounty for or running a competition for whoever could correctly guess both the score and the two teams involved in the Premier League final If you want to find potential Brexit voters Seems like a pretty good way to go about it But it doesn't speak to the efficacy of your underlying algorithm or use of data It's just a clever idea to select for mid 40-something white men and get them to give you their email address and some other Is there any way that we could measure the efficacy of it? I mean if ultimately what you're trying to do is influence votes and votes are not Public You know, we don't have we don't have records that you can look at we know X number of people from this area voted And we know how the the totals came out, but we don't know that this guy voted this way unless he tells you Can is it always gonna be just kind of that monorail sales pitch or Would it be possible to say look we you know through what we've done We moved to the popular vote, you know 0.0 whatever percent I Think it's difficult Maybe not impossible in certain cases, but you rarely get the chance to rerun an election while only tweaking one variable and In order to really drill down into the efficacy of this you'd need to be able to do that. I want to get to the The government's Response to this and and the the proposals that have been put forward to fix this problem but before we do this has provoked a lot of soul searching in Silicon Valley and then And and a lot of you know talk about privacy and and I don't know if it's related to This stuff or if it's related to the the recent changes in European privacy Regs, but like just over the last several days. I've gotten notices about our privacy policy from basically every internet service that I Have signed up for So this stuff is in the air what What are people in Silicon Valley seeing as like well sort of first? To Silicon Valley recognize this as a problem or or think this is a problem It might not be a problem, but do they think it's a problem and then what are their kind of self? enforced solutions that they imagine So when it comes to Silicon Valley's perception of their role in this I think it's even broader than simply a data privacy issue but a growing recognition that They will be treated and must behave as a sort of political actor their power has been recognized and now they have a bunch of people lining up at the door asking for various dispensations and Threatening different sorts of regulations if they don't get what they want So this is an emerging awareness when it comes to what Firms have done in response to this It's been fairly robust Particularly when it comes to this Facebook Russia question Russia used Facebook groups in many cases very large pages to spread their messages trying to come off as American citizens of different political bents they're requiring in order to run both Explicitly political ads, but also just any kind of more general issue ad that falls within Laundry list of political categories Or to run one of these large pages. These are pages with X number of followers and these are all like they were there were those America great or Things like this patriotic sounding Yeah, vague make America wonderful now kind of thing But going forward and this is supposed to be rolled out before the 2018 midterms in order to do any of that You'll need to verify your identity that you are an American citizen or have some form of government issued ID and You'll actually receive a code in the physical mail That you then punch in online and then you've been verified. This is what Facebook has announced. Yeah Shouldn't we be kind of flattered and grateful that Russia wants to make America great again? I Mean they're just trying to help out. Yeah, I agree. Yeah, you know, we can we could use some help So this this is like how recently did they announce this Facebook? This was pretty recent pretty recently It had been in the works for a time They're also and this will be fascinating for all of the DC political wonks going to keep track keep a publicly accessible Database of all political advertisements run by all campaigns So you will know you'll be able to go on and look at Both what might have been run to you But also adds designed to target very different sorts of people from yourself But this will be open and accessible to everyone. It makes sort of sneaky AB ad testing much more difficult I'm sure some people be rather frustrated about that but This is an action the government certainly couldn't take but that Facebook as a private company has decided to do What about on the broader just Creepy data mining and aggregation Level I Mean so one one question I guess would be is that stuff is the you know We're just going to gather data like all of the data about everything that you do on on our platform And also on every website that's you know interfacing with our platform and whatever else is that like necessary to the kind of very business model of These free mega platform sites could we have Facebook without Creepy Facebook data gathering You could if you wanted to pay for it whether that market exists. I'm not so sure how Real is the threat of regulation. Do you think I mean maybe not even in the near term Regulating Facebook or Twitter or something like this, but but in the long term too real We're in the as you and I have discussed privately we're in the baby steps of the internet and of social media and The world in 50 years will I assume social media is not going away due to the sort of human element of it You know what we will will we have to be constantly aware of sort of calls for regulation Because I'm thinking about comparing it to something like the FCC right where we had a fairness doctrine until 1987 that on the theory that the limited airwaves and if you just put one side of the of the Conversation up you had to put the other one because that's how we sort of sculpted our political speech Framework so people could be informed of both sides Do you think that there's a possibility that something like that or totally something different could come with Facebook another Yet to be seen social media. I I'm certainly concerned about the threat of regulation going forward Um CD a 230 has created a pretty good and was that you did pretty strong CD a 230 is an element of the communications decency act and pretty much the only element that survived later judicial review It prevents platforms or content hosts from being held liable for content posted by others If you have a hand in making the content doesn't apply to you, but as long as you are just Up or down voting what others create You know someone libel someone in your comments section They can't go after you they can go after the person who posted the libel, but you as a host are insulated and It's really the Substrate on which the modern Internet has been built Without it you couldn't run a platform like Facebook because you'd be sued out of existence However, well that's created a pretty strong presumptive norm In favor of allowing platforms to govern themselves you are seeing it nibbled at around the edges particularly in areas in which There's an ugliness to what may be happening on the Internet. We saw foster past recently which carved out a little section cutting away at these 230 protections for Sites that are seen to knowingly promote sex trafficking or prostitution Hard to be against an anti sex trafficking bill, but in its effect You begin to expose platforms and particularly up-and-coming platforms to liability for things frankly beyond their control On the advertising front as well It looks as though Facebook's move and Twitter as well to privately Rane in their political advertising markets have at least immediately Forstalled regulation, but some form of the Honest Ads Act will probably move forward in the future simply in an attempt to standardize Advertising rules between Broadcast print and these digital mediums. What does that say the Honest Ads Act and I've been a nutshell it is an effective expansion of Existing broadcast regulation to the Internet Now whether the power to do that really exists is Somewhat questionable because you aren't talking about a limited broadcast spectrum anymore But it does seem as though the political will to put something like that through is there What would the effect of that be I mean so on the one hand we could say if if government cracks down It's going to it's going to cripple social media, you know or the the foster Regulations meant a lot of sites were shutting down Turning off sections that that sex workers used And so we might say well, this is gonna this is gonna destroy everything but I guess I could see it it could also push us in the right direction not not in terms of I mean It's good that these things are being shut down, but the response to them so like that the sex workers launched a Federation of Mastodon which is a Decentralized Basically Twitter clone that you can run on multiple servers that can talk to each other So it's a decentralized Twitter and moved their conversation there And that's something that you know You might be able to shut down one instance of it But it can pop up somewhere else and you could you could build it in such a way that it couldn't be shut down that It's fully distributed and they're now back people are back to talking and there's hundreds of thousands of people or hundreds of thousands of hosts on it So it might be that the response to the government going after all these centralized services and say we're gonna regulate you is that people Go back to Decentralized services that can't be regulated which I think would be a good thing potentially though I'm not sure those decentralized services are as robust as we might hope for them to be That SW list Was recently bumped off of cloud flares DDoS protection which took it down it may have gotten back up on its feet now It's back up. Yeah. Um, well, that's good. What is the SW list again? Is that a white supremacy thing or no, no This is a decentralized social network. This is the this is the Twitter for sex workers Oh, okay. I think so it's called mastodon mastodon is the name of the underlying technology So anyone can set up a mastodon instance Like a you know, it's like installing kind of wordpress on your own server, but then it can talk to other instances as well So, but they're more vulnerable as you said then we might initially expect There's a thought that well, it's decentralized. So it's censorship proof You still need the DDoS protection And I'm sure some of that will be worked out as we go forward however, the other concern is That this regulation will cement the status of current market dominant platforms Facebook YouTube They've got a lot of legal clout. They've got huge war chests They can afford to work under some of this proposed regulation. It'll be more costly for them But they can bear that costs a New startup cannot when you're at the three guys in a garage stage you can't afford a legal team and I'm concerned that as a result of some of this upcoming regulation The next Facebook may just be strangled in its crib It would be like being stuck with Myspace back when when Zuck was building a Facebook in the garage If Myspace could have had these regulations. We might all be still friends with Tom, was that his name my space Tom? Yeah my space Tom. Yeah, so yeah, it's somewhat Trite looking back to see proposals to nationalize Myspace But people made that case then and it was so important that it needed to be nationalized and had that happened We would still have it And then I have seen people saying about Facebook too I mentioned earlier that the disconnect between the way that Congress responds to the revelations of data gathering and mining by American intelligence agencies and Facebook doing it and a lot of people Who are you know, they rightly are upset about the NSA gathering all of our data, but they're they're upset about Facebook to Because they see there's a lot of like these basically the same thing They're they're widespread surveillance and widespread surveillance is bad Is that the right way to look at it? Like is it is it conceptually does it make sense to think of what Facebook is doing as? Massive surveillance and should we be worried about it in anything approaching the way that we worry about it when the NSA is doing it It depends upon what your imagined threat is If it's this sort of social discipline, you might be more concerned about Facebook However, as far as we know Facebook data is not used to in Snowden's words put warheads on foreheads The NSA's data is So there is a difference there However, when it comes to how this Facebook data could be used down the road Well, legally if the NSA wants it, they're likely to be able to get access to it So the mere fact that it has been collected and made legible Can be concerning Why is this all about Facebook? Like Facebook does this stuff and it gathers massive amounts of data about us in order to sell ads and that's its business model But the business model of Google is to gather massive amounts of data about us in order to sell ads Twitter's business model. They don't gather quite as much data, but they gather a ton of data about us in order to sell ads Is there something So on the one hand is like is there something technically different about Facebook that makes it creepier or is there something? Culturally different about Facebook that gets people more upset. I think it's the latter When you think about Google it's somewhat difficult to connect your use of search or YouTube to Google ad words or their other advertising properties When it comes to Facebook, it's all occurring within the same walled garden You're reading your friends posts on Facebook. You're also receiving Advertisements from Facebook right alongside them so it's easier to think about it as a data harvesting and advertising entity within a social media space Whereas when we look at Google The way in which they collect information and then use it to sell ads is More opaque and distributed Of course, we can't For the same reason we discuss with my space We can't presume that Facebook will be around forever and their networking effects and all these things But but there's also the the intergenerational element. It's true. Yeah kids don't use Facebook It's what old people do it's underappreciated that for the most part Many of us online today all came online at the same time regardless of our ages however as new generations of So-called digital natives I'm laughing is this there's this Going around the office everyone was using the term digital natives and some of our colleagues were like digital native What is that digital native so so are you your will is a digital native? Are you digital native? I guess so okay On the border at least but Analog native it's true Most young people don't want to hang out in the same spaces as their parents You don't want your parents to see what you're up to with your friends, and I think you will see much more A age cohort based segregation between platforms Youth just learn how to use Facebook's post-privacy tools so then they can share their posts only with their friends and their parents can't see it It's still and if you're on there your mom's gonna want to be friends with you Well, she'd be friends with you but you stick her in a group, and then you say share with just this group Which is my homies I think I mean I won't speak for 15 year olds But but they seem to like more destructive Meaning like snapchat right it goes and then goes away, and they like things that are sort of demonstrating their lives so they can Film them and send them out to people. That's what I don't know That's what the kids are doing, but they're shifting to messaging that too, but that but so this all brings up So I mean political communication in the next decades Is I mean we're not gonna be able to predict what it is But it will be the main way that people form their attitudes about almost everything I mean they're like even the news networks They're there, you know their median age is whatever 60 years old or something like that No, 23 year old is going to start watching 7 o'clock news You know what in 10 years they're gonna get it through all these other things So we're gonna this conversation about how political opinion is formed and the interesting thing is that more it Targets on my going back to my theory of not having a good theory of the other side The more you're effectively targeted with just things you already agree with the more the other side will look Completely insane to you and then this question. I do think that there's something like a fairness doctrine That will be seriously discussed for social media in the next 20 years. I don't know if you have any thoughts on that will I Really couldn't hope to effectively comment on the 20-year horizon of the fairness doctrine. I will however point to I think one of the most interesting facets of This claim that Facebook has been censoring conservatives or other marginalized viewpoints The fact that These two women diamond and silk Who are political? E-celebrities they've risen to prominence by streaming Themselves discussing Donald Trump Were enough of a political concern to The Republican establishment that today they were invited to the Hill as witnesses To discuss how they may have been impacted by certain changes to Facebook's algorithm That's a deeply bizarre development and Speaks to a digital culture and digital political life That is increasingly eating real-world politics That that conservatives thinking that they were being censored on Facebook thing. I Have wondered about that because so the the claim is that our pages with our posts on our our You know Patriot nation whatever page aren't getting anywhere near the reach that they used to so we must be censored But all of this was happening at the same time that Facebook announced that they were basically reducing the reach of all Pages and so this is this is like one of my pet peeves in this space is that people Routinely don't understand how the technology works have no idea how the platforms work stumble across Something and then think that it affects everyone right but they just noticed it for the first time It's been going on forever, but they just noticed it for the first time and they assume that it was like directed by some engineer at them And so then they blow up and it's like this grand conspiracy When in fact it's like no this was just we changed our policies for every one six months ago And you just weren't paying attention You think conservatives are particularly bad at that. I think that by and large given the demographics Conservatives especially of the the mega sort are probably conservatively less tech savvy They they skew older who tend to be less tech savvy They skew to other demographics that are not going to be as tech savvy. So I think that does they're probably not as Media or online literate as mother group. They also might have more of a persecution complex particularly in relation to Bay Area Liberals When the daily coasts sees that its traffic is tanked after this algorithmic shift There isn't really a perceived grievance against them on behalf of That those who run and operate these platforms When it comes to conservatives, there's already this culture war tribal distance So I think it's just easier to impute that animus Whereas it wouldn't occur to the operator of a liberal page So people who use Facebook they begin this discussion talking about you know, well, we talked about Congress They don't seem to understand it or these things if someone's gonna learn something from this scandal putting that in scare quotes to About maybe things they didn't know about Facebook or you know, what what you know, what should they realize? Is a lesson from this in terms of Facebook and social media general the lesson that I Frequently look to when these stories come out is that everything is permanent online And there if I can jump for a moment from the broader Facebook Cambridge analytical Political scandal to an incident in the past week involving some blog posts that Joy Reed may or may not have written about a decade ago They were dredged up on the internet archive the Library of Congress had some copies She's claimed that they've been were hacked or manipulated in some sense, but it seems as though they just she attempted to memory-hold this stuff and as much as We heard in the hearings with Zuckerberg that well when you delete your page, it's it's gone forever That may be the case within Facebook servers, but anyone you were friends with might have saved a copy of your page or saved screenshots of things you'd posted and the extent to which everything online is Permanent so long as someone out there could be the smallest actor in the world Who wants to retain it? I Think that's still underappreciated and will remain underappreciated for a while. I guess looking forward we're Sitting in April right now as we record this it may come out in May and At the end of the year we have a congressional election And then two years later We're gonna have a presidential election and all of this stuff that you know if people are still Relitigating the 2016 campaign They're gonna get pretty hysterical about this stuff when it's the control of Congress or the next president What do you think We being both the the American electorate and how we engage with Facebook and Twitter and other things online and then also policymakers Should do about these concerns before we get to November I think we ought to keep a very close eye on How these platforms are used Coming into the midterms because all of the incentives to either misuse them in the case of Russia or Play up that misuse when it comes to a host of domestic political actors None of that has changed between 2016 and this coming November Russia in particular will have every incentive to Metal just enough or even claim that it has metal just enough that The American government will continue to attempt to hobble its Vital tech sector those who want to see more regulation of Online advertising and even speech well again have every incentive to play up the impacts of bad speech so policing these sorts of claims and Even if it's the cleanest election in history, you'll see a lot of them becomes a vitally important democratic duty Free thoughts is produced by test terrible if you enjoyed today's show Please rate and review us on iTunes, and if you'd like to learn more about libertarianism find us on the web at www.libertarianism.org


Hello, everybody. Welcome to this session titled lessons learned from a large-scale OpenStack deployment with Triplo. Let's jump into the session. So a little bit about myself. I'm Saishandar Malini. I work as a team lead in the Red Hat Performance and Scale R&D division. I'm focused on private cloud technologies like OpenStack as well as hybrid cloud technologies like OpenShift on bare metal. I'm based out of the greater Boston area in the US and in my free time I enjoy traveling and hiking. Over to my friend and colleague, Pradipta. Hi, everyone. I am Pradipta Sao, working as a senior software engineer in performance scale team at Red Hat. I'm focusing on technologies like OpenStack as computing and NFP and I'm based out of Pune, India and I'm loving to travel food and watching movies. All right. So like we mentioned, we are part of the performance and scale team. So what do we do? We are really on a mission to make sure that Red Hat OpenStack platform is the most scalable and best performing OpenStack distribution out there and because Red Hat is an upstream-first company by doing so we are also ensuring that all of our code and all of our fixes go into upstream and the community benefits from it. So overall end users have a better experience with OpenStack at scale. Also over the past few years, repeatedly talking to customers, OpenStack has matured to a point where it has a lot of functionality and customers are no longer really worried about the functionality but they're starting to expand their footprints in terms of number of nodes per cluster and so on and so forth and they're really looking for something that scales and scalability and performance has repeatedly come up as a priority for our customers. And when we talk about the platform scalability itself which is OpenStack scalability, you know, I think the first step to getting there is having an installer that is scalable because let's all agree. I think we all agree that we can't have a platform that scales without having an installer that scales and that's able to deploy a scalable platform in the first place. So over the past few years and few releases, we've been laser focused on making sure that OpenStack scales to numbers that have not been tested before. So where do we do all of this testing? Initially, you know, we were using partner labs to get to these high node counts because, you know, it's not easy to have hundreds and hundreds of bare metal nodes to do this testing. But, you know, starting in 2016, you know, we've realized that to shorten the feedback loop between the feedback that we get from scale testing and, you know, the product development itself, we need to build these capabilities in-house. So we invested time, energy, and money into building what we call the Red Hat Scale Lab, which is a lab in excess of 900 bare metal servers, you know, a combination of Dell and Supermicro bare metal nodes. This lab supports scale testing of all of Red Hat's products, including but not limited to OpenStack and OpenShift, of course. And, you know, the lab has really helped us accelerate our testing and our supported node counts and, you know, had a positive impact on both upstream and downstream OpenStack scalability and performance because this is where we do all of our testing and this is basically our playground. So because this talk is around, you know, building scalable OpenStack clouds with Triplo, let's take a quick overview of what Triplo is. And also, while doing that, let's all acknowledge the fact that OpenStack deployments are not the most straightforward thing to do because when you deploy OpenStack, you're basically not just installing software. It's deploying and architecting your cloud platform. So Triplo is a tool that helps you deploy and update and manage the lifecycle of your production-grade OpenStack cloud. So two important concepts in Triplo. You have something called an undercloud.  So the undercloud and overcloud and, you know, architecting and deploying and, you know, making sure your undercloud code scales is an important part of being able to deploy a scalable overcloud. So this is a quick snapshot of Triplo scale testing that we've done over the past few releases and years. Up until the Pyke release, we've hovered pretty much around the 300 node count. And up until Newton, we were relying on external partner labs. But you can see that starting with Pyke, we've moved all of our testing to our internal scale lab. And, you know, with the past couple of releases, we've achieved an excess of 500 compute nodes per cluster. And specifically with Train, which is an OpenStack 16.1 release, long-term release, we've achieved an excess of 700 compute nodes, all part of the same cluster. So let's look at the scale testing that we've done for our most recent release, which is Train. So I'll give a quick overview of the deployment before passing it off to Pradipta to take us forward. So the deployment methodology, like we've been discussing is Triplo. We're using Triplo as the installer. We have one bare metal undercloud node. We have three monolithic controllers that run all of OpenStack's services, like API and, you know, Nova Scheduler, so on and so forth, in an HA fashion. We have three Ceph storage nodes with four disk each and 712 compute nodes overall. So obviously we didn't deploy all 712 compute nodes at once. We slowly scaled up to 712 compute nodes over a period of time, you know, a couple of weeks. And also the Ceph storage cluster itself is pretty small because we're not really trying to improve Ceph scale here. We are more focused on the Ceph client side, which is the compute node and, you know, how many Ceph clients can the small Ceph cluster handle. The deployment was through Ironic and Pixie. We had about 14 composable roles overall. So one for compute, one for controllers, one for Ceph storage nodes and 12 different composable roles for the compute nodes because all of these compute nodes were from different manufacturers with different network configurations. So we really needed composable roles. So we were actually even scale testing in terms of the number of composable roles used in the deployment. Like I said, at the end of this exercise, we had about 712 Ceph clients looking at the undercloud and controller specs, you know, Skylake machines with 32 cores with hyper threading, so 64 logical cores. The controllers had a little bit more memory than the undercloud, but, you know, all these were pretty beefy machines. And, you know, looking at the Ceph storage specs, we have same Skylake machines, but in this case we have four 3TB NVMEs for the OSDs. So I will hand it off to Pradeep to take us forward from here. Thank you, Sai. Yeah, I think Sai has covered all this deployment type. Coming to the software specification, open stack version we are using trend release and the deployment methodology of OpenStack we use TRIPLO, which is based on the config download, Ansible best config download method. And for OpenStack performance testing tool, we are having PROBIT and RallyTool, which having various scenarios to execute to measure the OpenStack tenant workload. For monitoring site from end to end resource for undercloud and overcloud, we are using Grafana and Coletti as in installing on undercloud and overcloud nodes. For installation of OpenStack trend release, we use the base operating system Red Heart Linux 8.2. For networking back in OpenStack trend, we are using OVN, which is the default networking back in. For OpenStack control plane, we are using monolithic control plane, which is three controller and having high availability for functionality. For undercloud and overcloud, all the OpenStack services are containerized in trend release and it's all the container service managed by Podman. Next slide, please. Yeah, coming to the deployment workflow, as Sai already highlighted, that triple is having the basic principle of OpenStack on OpenStack. The first OpenStack services is installed on the as an installer, which is named as undercloud. So in the scale environment, we use undercloud as a bare middle node with the sizing of control plane IP, which is required for our 712 compute node and including controller and safe node. So we did not integrate any high advanced feature like Minion and routing functionality here. All the provisioning request is a layer-to-scenario. And the second, we applied the OVN patches, we aware of the OVN patches from our OVN team, like which is potentially fix the scale issues and also give us the stability in the scale environment. And as a prerequisite from the triple side, we created a heat template for the composable role for our controller, safe and compute nodes from the various hardware type and also we created profiles and flavors for different hardware type for further deployment. And once we have done all this template and profile flavor, we are trying a basic overcloud deployment with minimal node count of three controller, three safe node and one compute node. And we have successfully deployed that and after the deployment of basic overcloud, we continue to register the remaining available node in the lab allocation and we will continue to do further scale out of compute, available compute node. And in this way we achieve 700 compute node using a 12 composable role. And after all the deploy, like after the scale deployment, we have done some exercise in the control plane scalability as well for the key services of NOVA, Neutron and Cinder. For all this exercise we have monitoring the environment through Grafana and we capture all these document steps which is required for the scale testing and we capture the debug methodology and also we file multiple bugs for further fixation in the future and for the enhancement and fixation in off-stream site as well. Yeah, coming to the highlights there is no regression in scalability in OpenStack Relays compared to OpenStack Queen because of this, TRIPLO has introduced with the Ansible Post-Based Model config download which is significantly reduced the undercloud resource utilization and due to that we don't need to bump off any worker thread and default timeout value for Keystone like the key resources of overcloud for Keystone, Mistral and Jacker and post introduction of Ansible Limit feature also help us to minimize the deployment time because we can selectively choose which node we need to be configured overcloud services and the next highlight is Heat Engine is consumed the lower memory footprint compared to previous OpenStack Relays as we significantly optimize the Heat Engine from the Pike Relays and now we have it use the lower memory footprint in OpenStack Train and after the deployment as we did some exercise in control play like control plane scalability and performance for 2000 VM creation and volume attachment we did not also observe any regression and the next highlight the RabbitMQ is very stable in OpenStack Train Relays we did not observe any regression and did not find any kind of exponential spying in performance spike in our monitoring tool and OVN also is getting matured to scale the ascent to 700 plus compute node as a default neutron backend and the measured thing is the Ansible config download which is more faster comparing to the previous OpenStack release like Queen for an example using the limit option we tried 100 node deployment and it took around less than 2 hours which is quite faster comparing to the previous OpenStack releases here is the one snapshot of Heat Engine resource consumption at 500 compute node the Y axis is considering as the memory consumption and the X axis is the time duration of the Heat Engine uses for the scale activity so if you see that a Heat Engine is gradually increase the memory consumption in when we are batch of compute node for scale out activity and in 500 compute node we observe it still having 8GB of memory utilization where the thread utilization is having 25 core and it's comparing to the previous OpenStack release it's less and which is the good positive for the OpenStack trend release and it's a yeah coming to the 700 node like after 700 node deployment in we observe like Heat Engine which has configured with the 24 worker thread it consume near to 50GB of RSS memory and for further validation we tried Heat Engine restart the Heat Engine which bring down the memory to 3GB and which it looks like as a linear way and it slowly increase like the memory when we do the scale out operation for the remaining node and in the entire validation we did not observe any memory in Heat side. As we did some extensive control plane performance scale after the scale deployment in 700 node we use the rally scenario to execute 200VN request with across the compute node and with the various scenario like the nova boot server volume creation and attaching volume meet and list out the server and volume and attaching a network port so in the entire operation like if you see the graph there is no exponential spike over here and all this resource utilization is in a linear way and the best part is the nova boot server it looks like very less time like 13 seconds to boot a instance so which is like better comparing to the previous open stack releases next slide please yeah of course there are some challenges I wanted to highlight all this issues which we have updated it has already been fixed in upstream side and also be documented in documented so when we did the test the end user experience is little rough like from the deployment perspective the first challenges was we faced like over cloud deployment command with limit did not reduce the deployment time for safe Ansible as the safe Ansible script as kickoff by the config download and due to that we came up with some manual intervention which is support for the safe Ansible script with the limit parameter for specific nodes so that we can continue get faster deployment and we can reduce the deployment time and the second challenge is the vical expression consumed too much memory when we increase the 200 node count 200 node count and hit stack got failed with this error and it because of the new vical expression has integrated and due to and it's a so for that we have to increase the memory count but yeah we raise a upstream bug to optimize the vical expression that we can continue with the default memory node count in when we scale 200 plus node and the next challenges deployment server blacklist which is usually use this triple parameter to blacklist the specific or faulty node in the over cloud environment so we tried the scenario in using the limit parameter Ansible limit feature but there is an impact we observe like all this Nova compute services goes down and due to some additional Ansible playbook so our deployment framework team has addressed this issue and it has already been fixed in upstream side we tried to document the steps which is applicable for operators so that they can use this parameter in a proper way in last scale environment so continuing the challenges I will hand over to Sai and yeah thank you yeah so I'll continue with challenges here so one of the things that we observed was the default Ansible fork count that is set by the config download playbooks was 10 times multiplied by the number of processes on the machine that is the under cloud and in our case it was 64 core machine so what happened was the fork count was being set to 640 which is not a great idea because you are obviously going to be bound by the memory here if you have so many forks so we raised an issue and we worked with upstream to lower this fork count we also did extensive profiling of Ansible tasks while the installer was running so we were able to identify tasks that were specifically taking more CPU or memory and based on that some refactoring of the task was done also when it comes to OVN and scalability we identified that the OVSDB bundle was not replying to the liveness probe from pacemaker when it was really busy and when it was taking a long time to process its loop so what we did was we bumped up the default probe interval for pacemaker so that it does not do a failover when not needed also we had to bump up a couple of timeouts here like OVN remote probe interval and OVN open flow probe interval on the compute nodes to give OVN controller on the compute nodes enough breathing space when there is a lot of churn happening in the environment like for example when you boot 2000 VMs at once there is a lot of churn that happens in the environment and OVN controller gets really busy so we identified these tunings and we documented these so coming to the outcomes of this scale testing like any scale testing activity we want to identify issues before customers run into them and fix these issues so we came up with bugs that identified issues and we also came up with tunings that work for scale and in cases where it made sense we actually pushed these tunings as defaults upstream so you get this out of the box for free without any performance engineering would do we also feel confident that we've been able to provide a better end user experience at scale as a result of this testing and as I mentioned previously we did extensive profiling of Ansible tasks during installation that really resulted in some changes architecturally as well as refactoring some of the existing Ansible tasks to make sure that Ansible consumes a lower footprint on the under cloud as well as Ansible runs faster and as Pradipta went over and I also did in my previous slide we filed close to 15 bugs and these are the high level issues I'm not going to go into detail with each of these but just wanted you to know that we filed bugs, worked with developers closely and got these fixed so that customers have better experience at the end of the day the last part of this presentation I want to walk you through some of the best practices and lessons learned doing this multiple times over several years over several releases what are some of the best practices and recommendations when you have to do a triple deployment at scale so one of the first things I want to highlight is always try to use a bare metal node for under cloud as tempting as it is to use a VM we have to keep in mind that the under cloud is an integral component of triple low you're not done with it as soon as you finish your deployment you have to use it for upgrades and updates so the under cloud is a really crucial component and it's best to use bare metal so that you don't run into problems later on even if your initial stack deploy is fine I would also recommend using a 10 gig provisioning network so that the provisioning is faster and you're not bottlenecked if you use a 1 gig card also you have to plan your under cloud for scale right at the moment when you deploy things like the control plane subnet range are hard to change once the initial deployment has been done so always deploy and configure for scale I think it's also recommended to use memcached for caching with keystone and heat on the under cloud for better performance and you should get this for as a default right now as a result of a testing we push these as defaults upstream if you're not using telemetry on the under cloud it's better to set the notification driver on the under cloud services to know up so that it eases some pressure on the under cloud rabbit mq and usually I think the first bottleneck people might hit is when you scale past 250 nodes per cluster you start seeing heat stack update failures although you don't see them as often now because a lot of improvements have been done it could be due to insufficient keystone heat engine workers so watch out for that but again as defaults we have better work accounts now out of the box then Ironic has a lot of bare metal nodes to manage in this case 700 plus compute nodes the conductor can get a little busy in terms of CPU usage just getting power state and polling that so we've seen that pumping out the sync power state interval to a higher number which is the number of seconds here helps to reduce the number of CPU spikes also when we add you know a few compute nodes to an already existing large over cloud cluster I think it's recommended it's highly recommended to either use the skip deploy identifier flag in the case of older releases which use heat and over skillet config for configuration or the dash dash limit flag in the case of newer releases which use ansible config download to cut down the deployment time if you're not doing any other changes you're basically just adding a few compute nodes this is the best way to cut down the deployment time and make sure that ansible or in a puppet doesn't run on all of the existing nodes and only runs on the new nodes to save time also I recommend scaling up compute nodes in batches if possible because that will help you identify and root cause issues and they're any much faster based on our experience we can also say that the monolithic control plane itself can handle both control plane load as well as you know the scale of the cluster if the controller nodes are sized properly so sizing becomes really crucial here and yeah we are almost at the end of this presentation and I just want to leave you with a couple of links to some of the blogs we've written about scale testing open stack 13 and open stack 16.1 which are queens and trained and respectively upstream and with that thank you for watching our presentation thank you


Okay, great. Well, thank you all of the cameras back there. Thank you all for participating in this virtually. Thank you to those of you who are able to be here in the room. We just wrapped up a terrific summit, the second open source software security summit taking place here in Washington DC. This was a follow up to a meeting that was held virtually in January 13, where a number of us came together along with folks at the National Security Council, OCD and other government agencies to talk about the state of security and the open source landscape and really identify some key problems. Since that time, many of us in the open SSF community and beyond work together on a plan, a plan that we debuted today here at the event, along with our stakeholder from open SSF but also with a broader set of industry partners, open source community partners, and of course with our friends in the White House and in government. This plan is now available on the website. The details of it are there in the plan. It represents a collective effort against 10 different targets that we've identified as meaningful or meaningful work could actually be applied to make a substantial improvement in the state of open source security. They are, they should be viewed as a first draft. In fact, I think we've labeled it version zero dot nine dot one, but with some specific goals and some specific approaches to addressing those key problems. We will continue to work with the stakeholders here. Now that it's public will also be looking for new participants in further development of that plan. It really I think represents a great flag in the ground as we said, I for further evolution and state of security in open source software. We had at the meeting today, I believe it was 80 different individuals from over 50 different organization organizations of all sorts, representing all sorts of different viewpoints and priorities. So much that we were able to come together around and and meaningfully, I believe come together in a consensus around the plan that we put together. We're also pleased to announce that a subset of organizations, I decided to make the first set of financial pledges towards the plan. Let me be clear the plan called for about $150 million in spending over two years across those 10 different lines of effort. That's a meaningful amount. We realize that is an amount that from some degrees as much more than anyone open source developer has or even even most open source projects. But when you compare to the cost of remediating a major vulnerability out there like we've seen in the last few years is a drop in the bucket of a very small ounce of prevention to spend to get to many, many pounds of cure. Okay, and we are able to happy to announce today that we've got the first set of pledges towards that plan. We can announce $30 million in funding from a set of partners to be allocated as time goes on across those streams as we refine the plans as we converge on specific milestones and possible achievements, but these are pledges that are incredibly meaningful. Those pledges come from Amazon, Intel, VMware, Ericsson, and Amazon, VMware, Intel, and Google and Microsoft. Thank you. We also does include $10 million of that amount is from Amazon as well. So, we're incredibly thankful. I'm so nervous that I'm forgetting my own name, let alone a member so apologies for that. And we will work with those as well as many others out there that are learning of this plan that we aim to collaborate with to refine each of those different elements of the plan and move forward on so incredibly thankful to those partners. Yeah, I'm giving you my take on the day. I'm a little biased as kind of the ringmaster of the circus, but I really wanted to also open the door to the chairwoman of our board at the OpenSSF, Jamie Thomas. Jamie, would you like to introduce yourself better than I could and then tell the world. Thank you, Brian. I'm Jamie Thomas and I represent IBM at the OpenSSF and I'm really pleased to be here. First of all, I'd like to thank Brian Bellendorf and Jim and all of the OpenSSF members and of course the government for their participation today I think it was a really a strong example of collaboration and what we can accomplish going forward. I'd really like to thank as well Deputy National Security Advisor Ann Newburger and her staff for being here all day with us and contributing in a meaningful way. I think this exhibits our intent as an industry to really create an execution plan to make a big difference here in terms of cyber security going forward. Clearly, there's a lot of execution details that we have to work out, but I think it was a great step forward and truly just really want to commend the collaboration across all the participants today. Great. Thank you, Jamie. And Sarah Navadi, Sarah represents Microsoft and sorry, you and Microsoft have been a tremendous partner for the OpenSSF. You helped with the January 13th meeting, helped in putting us together as well. Did you want to share a take on today's event? I'd be happy to. So I'm Sarah Navadi and I lead open source strategy for Microsoft, and I am very happy to see what we've done here today. It's bringing together the community excitement with the constraints of industry and the constraints of government and finding a path to development through all of those things. And then to do that together. And even just finding the common language in that 50 page document that we pulled together as a starting point. It has been a huge amount of effort and has been worth all of the time that we spent on it. And we will see, we will see rewards from this in our industry in our government and and in our collaborations going forward. We have as many companies as we do more than 30 now, right? Working on this and we would welcome any others of you. I'm also very much looking forward to taking this plan that we've started we've outlined and drafted, and getting more feedback on it and more people engaged with it to drive it across the whole industry. Last comment, I want to just pass the mic to my boss, the executive director for the Lenox foundation. This has been a passion of yours for a few years now. Can you comment on the moment here and kind of what happened in the meeting but putting that in the context of the last 20 years. I've been working in the open source community for almost two decades and in that period of time we've had multiple cases where a vulnerability in an open source component has posed, you know, dramatic risk to a broad set of society. You know, Heartbleed in 2014. Most recently, Logforge, you know, really put us all at risk and we've all spent a lot of time remediating these things. In this period, we have systematically tried to get help to the hundreds of thousands of open source developers who are out there, and to the leaders who are responsible for critical components of the open source supply chain to help improve the baseline of security. And today is one of the first times I've seen a actionable plan with concrete goals, but most importantly, an industry will to offer that help in a meaningful way. And we're in the first five minutes of a long game. The urgency here could not be greater adversaries are getting more sophisticated supply chain attacks are happening more often and cyber conflict is escalating around the globe. But just, I couldn't be proud of the work that Brian and the team have done, but also just at every level in the organizations that have convened here a serious commitment to bring help to all of the developers globally who write the code that makes up 80 90% of all the technology and services, most of us rely upon every day. It is a serious milestone. Having the Biden administration deputy secretary new burger activity, a new burger advisor new burger, helping with this, you know, working in coordination with the public sector and this great private coalition is just incredibly gratifying for someone who's been in this world for quite some time. So we are still working on the readout document from this that'll include quotes from many of the participants were actually scrambling to collect those in real time. Between the end of that meeting at the beginning of this, we will get that to you very soon. We're just closing up the loop on many of those will get that out but also as a press release over the next few hours. So, I, we know there's some some gaps perhaps before we get that out, but I again, really, we're excited to have hosted this meeting excited to hold this committee together. I'm very thrilled by the validation that has come from the series of pledges, but but beyond that, we know that the industry and the public sector are really coming together to address this in fact it's really nice to see this compliment the hearing side of the House Committee on science and technology hosted, focusing exactly on this topic on open source software security, and to see the enthusiasm amongst the policymakers for asking what can we do to help how do we. What's the investments, whatever the investments we can to to be that ounce of prevention was really gratifying as I mentioned to the room 2009 me when I came here first to work at the White House in the Office of Science and Policy, talking about open source software as a public good that that person's brain is blown by just how far the conversation is advanced here in DC. And I'll end with, we are eager to engage beyond the United States as well. We've, we're looking forward to conversations with other other public sector actors around the world about how to align efforts how to build upon each other's work and how to really open source offer is a global phenomenon. We're really eager to work with everybody who finds residents with the plan before. So I think with that, I'm looking at my staff to see are there any other messages that I forgot anything worth getting out and I do not have a mouse to be able to see what the questions are. Is there a way. Okay, why don't we start with this. Let's start with the first one. Sean Kerner asked the question, what was the biggest surprise for you, Brian, in the hearings yesterday. Well, as I mentioned the fact that the opening statements from the committee members were show that they and their staffs have actually recognized both the role that open source companies in critical infrastructure, and the opportunity to turn some modest investments actually into outsized returns in the former produced risk. What was great is that allowed me to cut my oral testimony in half and actually fit it into the five minute window, because they said what I usually feel like I have to say, in making that case. I think there is also the funny moment when I mentioned I'd rather use open source software that bugs that were found and fixed than one that hadn't. And one of the committee members asked, I hear you right, are you really rather use soccer with more holes. And I said to say, and fixed because that represents not only software that's being used but also software that's being addressed. So that is actually valuable enough that people are fixing it and a process that is mature enough to get those fixes out there to the world. But let me turn to other members here was there reflecting on the testimony yesterday. And maybe the themes of today, if I don't know if I had a chance to see. I haven't been able to watch the testimony yet. Not not a problem. But I, but really, you know, many of the themes came up of where to invest in education and scanning of code to get the distribution points and third party audits. And what I would say today is just the unfettered participation, if you will, I think everyone felt very comfortable and stating their, their concerns or their recommendations from their point of view. And I think it was the those unique points of view will help us fine tune the execution plan and be much more effective. So I think that was very good. Very good to see. Thank you. Sorry to interrupt Brian. Let's just give a little announcement because we have some virtual viewers here today. Great. So we've got, first of all, thank you everyone for joining us here today. We've got some journalists and members of the press attending virtually and some people here in person, and we'd love to feel all your questions. So first of all, to the people attending virtually over zoom, please submit your questions through the live q amp a function. We're going to go through them, we're going to promote you to a panelist and give you an opportunity to ask them. We're going to give you some time to submit those questions. And we'd like to start by fielding questions from people in the room. Would anybody here like to ask a question to the panelists. You feel like you've come away, you know, just a broad document, lots in it. Is there one thing that is the most important strategy for changing or making, you know, open source more secure. I mean, what is the news I guess what do you consider the newsiest aspect. Well it was interesting to me how often in our conversation today, as we were talking about one stream that it came up well this actually would benefit from a resonator depend upon even success with the other stream. This is really a portfolio approach. Yeah, in some ways you're asking me to pick which of my children are my favorites. So, I apologize for that but I, it really is, it requires a cohesive effort because there's not one root cause or one root I'm trying to address them all and it's industry recognize is that I think the public sector partners recognize that as well. It was nice on the one year anniversary of the White House Executive Order 14 to wait to have a fair bit of our conversation be about software bill of materials and making that meaningfully adopted. What does it take to actually not just come up with a good spec, but get it used by the open source community. I think that's been a large part of our work together. And it was great to see even just the interest in that and reflection of that. Yeah, I mean I think the big news is similar like a year after the executive order on cybersecurity, we're here now with an actionable plan, and what I find most compelling is just the industry will bow. You know, folks from Google saying, we have people that we're going to put on this to go and fix vulnerabilities, you know, folks like Amazon committing financial resources against this plan. Many of the folks in this room also committing, you know, both the expertise which in many cases around cybersecurity matters only rely resides in industry, but also financial commitments as well. We've never seen that amount of unified will to significantly raise the security baseline for us collectively and I think that's just an incredible accomplishment. You know, only six months after, you know, we all suffered from a major open source vulnerability, the White House, and, you know, Ann Newburger brought us here. And now we really have a plan that is already being taken into action, but now has significant additional resources to carry it forward in a meaningful way that that's a big milestone. That's something if you'll give me a chance, Jamie. I think that the big news for us is that we are looking at this from multiple different perspectives and while we have 10 streams we don't have to pick a favorite child because industry needs some things done government needs others, and we're going to work on funding all of it in some balance. Yeah, the only thing I would add is I think this really ensures that we all understand the importance of open source, how productive it's allowed us to be. And our obligations obligation to consider the implications of using it right so I think that was a fundamental understanding out of the first two meetings is that open source has allowed us to all be productive, but we have an important understanding in making sure it is secure, and that all of our downstream clients get the benefits of that security. So, can't agree more with the comments from Brian and others that these things kind of all come together to ensure this open source community can be effective going forward and that it can be secure at the same time. Great, thank you. Any other questions from people in the room. Great well we'll move on to our virtual participants. There's a question from Eric Geller on the on the screen of the right. So Eric Geller we're going to promote you to panelists that you can ask your question. Yeah, if that's possible or I could just read this question. Hey everyone. Thanks. Appreciate you taking my question. What is the timeline for allocating this initial 30 million has any of it been earmarked for any specific purpose or is it all up for grabs and did any of the other companies say that they might be able to make additional pledges in the near term. Um, the best way to think of it I think is as a venture capital fund, right, where here's a set of 10 investment worthy targets streams. Each one of them has a business plan. Each one of them will have to justify the investment that 30 million and pledges represents a fund that will spool out we expect over the course of a couple of years, but it's the beginnings really what hopefully will be a larger fund to cover the $150 million that we identified. Now, you know, as the plans evolve as we find ways to save money as we adjust the targets to be available funding will right size it to the opportunity. But this is by no means a final funding this is the first down payment think of it that way or down first tranche of investments into the fund. So it's really going to though be up to the funders as we work through the plans to say this stream is good. This one is now ready. Let's come in into this and push it forward and some of these activities are already begun. Yeah, I think it's also important to understand where money helps and we're also time and expertise helps from the organizations that are represented in the room today. You know, for example, just doing an audit, a third party audit of critical open source code bases is something where you need financial resources to pay an audit firm to do that. Once you find problems you also need assistance from expertise and developers who are very sophisticated to go actually fix those problems. In some cases that will come from organizations who have that expertise in house that are allowing their employees to spend time remediating these vulnerabilities that are found in the audience. So what you'll see is a blend of both financial resources but then hands on assistance from subject matter experts, and it's the combination of those two things that makes for the highest impact in the shortest period of time. Yeah, I can agree more Jim I mean we I like this VC analogy I think it's a great ongoing VC fund, but it is an obligation of all of us who have these skills to make sure that we're contributing individuals and their expertise to the open source community open source projects and Sarah had a great comment in the meeting that I'll just give her credit for is that we all have to recognize those individuals in our enterprises for their contributions to open source. And so I do think that's an obligation that we have to take on, because that's how we encourage them to do this and not just quote volunteer their evenings for this, but make it a part of their day job. And I'll add before we move on to the next question we internally pulled our membership at open SSF, the leaders in the community and asked how much money are you spending today on securing open source software I don't mean by giving away your product free I mean I actually sitting down and doing some of the kinds of hard work we talked about here, being a part of existing open SSF efforts or efforts across other parts of the open source landscape. And even just with a few responses to that we quickly came up above $100 million in direct spend and almost 100 full time engineers. And so this is an industry that's taking this quite seriously already this funding and the resources that might come from it are additive to that effort. And then also we're very eager to open the door beyond open SSF participants and and look for other sources of funding to help complement this this initial trench. I think we're ready for the next question. Thank you. Sean has asked us to read his question out. Okay. How are the targets and the new plan different from some of the problems the Linux foundations core infrastructure initiative CII tried to solve years ago after Heartbleed. I mean I can. So I started the core infrastructure initiative after Heartbleed, essentially to get direct financial support to the maintainers of open SSL who at the time, really, you know, we're doing this as a very, very small group of developers in a very highly specialized area of photography. And that was a case where just supporting a small set of individuals to do some work on critical projects that they really had a unique understanding of was helpful in the short run. It became very clear to us and what this work builds upon is that that alone is just not sufficient that you have to provide a set of resources that that include training of developers about how to write secure code in the first place tools that allow them to do better testing, you know, a better DevSec ops, you know, set of tools so that they can release code securely. And then the complexity of the overall software supply chain was not as difficult as it is to manage today. The explosion between 2014 and 2022 of small reusable components that have become the building block of modern software and that are distributed through different package management systems has created a level of complexity that's extremely difficult. This plan is very comprehensive direct support for developers, seconding engineers to solve problems, providing audit of code bases so that we can find voter abilities from a third set of eyes, going to the, you know, sort of friction points in the supply chain where package to use additional security, both tools, time and resources to improve just the ability to do package signing on distribution of software components, etc, etc. It is what is very different here this is far more comprehensive. And the final thing I would say which I said originally is, it's bought into at a far more senior level from government and from industry, and anytime in the history of open source this threat is immediate. It is complex. And, and people here are taking it very, very seriously. Thank you. A next question is from James Rundle at Wall Street Journal. James you want to meet yourself. Yeah, hey good afternoon everyone thank you for doing this really appreciate it. So just noting that there's been a pretty significant commitment from the private sector, and also taking on board what Jamie said earlier about our new burger spending a lot of time with you guys today and her team. What support will the US government provide for implementing this plan moving forward have you any commitments from them in terms of skills and expertise in terms of maybe leveraging their procurement power I know there's been talked about that in the past. Good yeah. So, yeah, so there is no part of that fundraising dollar amount that comes from the US government or any others and I just want to be clear we were not here to fundraise from the government. We are aware of interesting wins on Capitol Hill about around investment in this space we're hoping that they and the administration can understand what we're doing. The types of targets that we've chosen and what we think is required to hit those and that may serve to inspire certain actions or the like. We do hope to make sure that what we do is additive to existing efforts. We've done an entire agency effort inside the White House, across on CD OMB and missed and others around complimentary efforts focused on S bonds and other parts of the supply chain. Part of that was a conversation today about ways that we can be supportive of each other's work. We do not, you know, did not in this in this plan anticipate needing to go directly to government get funding for any of it to be successful, nor is it a part of the $30 million number. But we're all very hopeful this is a plan that benefits everybody, you know, and government is a major user of open source software, and it's starting to create open source software and distribute that as well. Yeah, so we're, we're, we think we have a lot of alignment in terms of interest and we're going to see the public sector get involved in elements of the plan. Yeah, I would agree with that I think one of the highest value things that, you know, get new burgers brought to the table is the convening function and leadership that's required. So I think, in the meeting we talked about, you know, you can't go fire hundreds of thousands of developers if they're not right to secure code it requires leadership, it requires someone to convene and to help us create a culture of security in the open source software supply chain, and in the source and the White House is definitely, you know, demonstrate that kind of leadership. The other thing that's important is there are already things going on in the government around cybersecurity that we want to make sure we align with in order to one get faster time to benefit and to reduce duplicative effort. And then finally, I'll just repeat again. A lot of the expertise around cybersecurity for this particular aspect of it resides in private sector, and private sector is stepping up in partnership with government to provide that expertise in a meaningful way. So just that that public private partnership and coordination is critical to getting the quickest time to impact from this particular effort. Okay, thanks. Thanks, James. The next question is to Kyle all back from protocol. Yes, hello, thank you. Just wondering if you could say a little more on S bombs. You know, how does this advance S bombs what what new things does this do for that effort. Thank you. All right, well, you know it is, as I mentioned the one year anniversary of executive order 1408 which really identified software bill materials as a key enabling technology to provide greater traceability to software open source or not in supply chains and in hosted software and that sort of thing. So coming up with a spec specification coming up with a file format isn't enough. You need to get this technology adopted, and you can create demand for it through things like curament which is what the executive order called for. But if it's at the tail end of the supply chain, then certain folks are going to hold it and make that a competitive advantage. What I believe needs to happen in the open source space is to have those that all that S bomb activity the traceability the generation of these bills of material as they flow through to move as far upstream as they can right to get open source developers who operate at the library level component level, you know way at the beginning of that chain to understand the value of it to be interested enough to to lift a finger to integrate it into their built systems. They'll only do that if it's easy. They'll only do that if there's if it's a part of the tooling that they use right if it helps them get their code to more people right so the plan that we put forward calls for investment into libraries and the technology that make adoption easier that go to key open source projects and embedded into their built systems that builds a reusable library focused on a certain approach but one that we believe is compatible with the major different structures out there, and builds bridges between some many of the different distribution systems that are out there today and artifact inventory kinds of systems so that's the key part of this stream it's very nerdy. It made some hard technical choices. There are all these dreams are very nerdy I apologize for that but I made some hard choices that are about making sure for the money we can spend. And it really in the best of open source is about building these components that then we hope gets get reused and and and enable a lot of other change across the entirety of the software supply chain. Can I add one thing. So one of the other things that is critical in this as bomb work. What I'm going to make that the case is this is a request from industry and enterprises for this to meet a government specification. A lot of our open source developers are accidentally famous. So we really need to make this as bomb generation super simple so that it is not putting toil upon the open source developers that we already depend on so much, and get much leverage for the amount that we spend contributing to open source as as as enterprises so we very much need to make sure that this is not putting additional toil on those open source developers and that we are bringing our industry weight to help and make it simple by using those choke choke points wrong wrong work but those those aggregation points key points of leverage. So one thing as well about not creating more of a source developers keep part of the plan across the different streams. Yeah, the easiest way to think about it is to understand how software flows. It comes from a developer's mind to a version control system to a package manager to a build system to a consumer. That's the simplified view of the world there's certainly a much more complex way to think about it. Another thing is to build software at every one of those points that automates the liquidity, a software bill of material metadata, so that it seamlessly flows. The outcome here if we can build this software will be a high degree of software bill of materials liquidity in data across the entire supply chain, that is the end goal we're looking for. I just add in and I think Jim you make a really important point. Number one action on the list today was education, but in fact we'll probably never educate everyone fully on the topic of security because it evolves every day. So automation and the instantiation of automation through things like this, so I'm actually critical and codifying the education for the open source communities and those that consume open source. I have a question from John Greg at the record. Are there any log for JS issues on the horizon. What projects are key to software that are of concern. Well, we published a report with Harvard called the Harvard census to which took a look at basically systematically across all open source software and try to identify pieces that are most critical pieces that the most other pieces of software depend upon and play perhaps a hidden role that maybe don't get the light of day that that they should in terms of attention to their their software quality or to the other kinds of processes that could lend themselves to to greater security. So I draw his attention to that as a way to kind of try to understand where where the potential risks are and we certainly believe that's been useful to certain organizations to help bolster the defenses by trying to identify these packages. The second thing to say is, you know, none of us have omniscience and can tell you where the next log for Jay will come from, except that, you know, there are there are vulnerabilities found every day. And even just in the Linux kernel, I think every week there's a release that addresses a couple of things and an important point here is software will never be perfect. You know, the only software that doesn't have any bugs in it is the software with no users. And so what's important is, how do you find them before the bad actors, how do you get them fixed as quickly as possible. And then how do you get that fix permeated out there into the rest of the world. And I think systematically you do that you can build resiliency and a response capacity for whenever the next major vulnerability hits. And I think that's an interesting point in that earlier in the day where you said, you were surprised by so many different pieces of software that said they weren't vulnerable to the log for Jay issue, because they had not even upgraded to the up to the version that was that was at risk for clarity. Yeah, the log for Jay two dot X was the one that had the vulnerability log for Jay one dot X, which had been out of support for five years and had lots of unknown vulnerabilities and many organizations said we're not we're an open source projects by the way. Not to get anybody off the hook here. And what it brought out was, we simply are not good even within the open source community about updating our dependencies clearing out technical debt. And this is something that we believe any of these streams will help help address. This is exactly the proliferation of the fix is the is another whole problem. Yeah, exactly. Thank you. We have a question from David Jones at cybersecurity dive. David, would you like to ask your question. Okay, we'll read your question David. What is being done to make sure the open source community is properly compensated over the long term beyond these initial investments. I guess that the, you know, for a lot of large scale open source projects such as Linux kernel projects like Kubernetes and others. You have a pretty robust economy of, in many cases, professional developers who work on those projects because so many organizations use them for their fundamental products and services so those companies. And the upstream of the source project and you have sort of a virtuous cycle of investment in those larger scale well supported of the source projects where things break down is the intersection of critical to society, and often ignored and underfunded. In those cases, unlike the consistent pattern of a sort of a virtual cycle of goodness and the big projects. Those individual projects tend to be very told story ask they are each sort of uniquely unhappy in their own ways it could be that there are people looking at it, but they just are missing certain security variables. It could be that there's a key maintainer that just suddenly becomes famous because that code is so popular and everyone's asking them for feature improvements and bug fixes.  There are ways to fix better aligning incentives with open source developers. One, some research that we've done shows that individuals would love to get more time from their employers to work on those critical open source projects. Understanding what those projects are in terms of criticality and then working with employers to give them more time to do that work as a maintainer, certainly is something that we can do. This is where it's just direct funding to developers who need that help in order to continue to maintain these critical projects. Those are things that we should look at to point is, there's no single solution. Much of the research that we've done also shows that many of the developers that are working on these projects do it for recognition for being a part of something bigger than themselves. And again, that's where an employer who is working with those individuals should recognize them to Sarah's point and provide them time in their regular day job to go work on those things. So it's a complex set of financial decisions to make sure that we align incentives correctly. Money is one of them, but it's certainly by no means the only thing. Yeah, I think we also learn from log for Jay, which has been out there for about 21 years is that often our developers gravitate towards the viewer most glamorous open source projects, but we still have an obligation to serve those most used projects or which will be more like that log for Jay's. So one of the things I know that we're all assessing are our most used projects we've certainly done that. In the case of IBM and committed more resources to those most used projects, which means we have to recognize those developers as Jim has stated. Thank you everybody for asking your questions so far we've exhausted all the questions from our virtual reporters are there any last questions in the room before we close off. No. All right, well, I'll leave it to the panel. Thank you everybody. Thank you. Thank you. Thanks everybody.