Hi everyone. Thank you so much for joining us today. My name is Bandy Singh and I've been spearheading OTI's work related to algorithmic fairness, accountability and transparency. Over the past year, OTI has published a series of four reports which look at how internet platforms use algorithmic decision making for a range of purposes, including content moderation, the ranking of content in search results and newsfeed, ad targeting and delivery and when making recommendations to users. Today I'm very excited to be joined by a great lineup of panelists to discuss the subject of our third report, which is how we can promote greater fairness, accountability and transparency around the use of algorithmic decision making and ad targeting and delivery systems. We will have some time at the end for our panelists to answer questions. So if you have any questions, please use the question and answer functions and we will do our best to get back to you. So now I'd like to briefly introduce our panelists. First we have Joe Westby, who was a researcher on technology and human rights at Amnesty International in the United Kingdom, where he focuses on the human rights implications of big data, AI and the power of tech. He is currently leading Amnesty's emerging program of work tracking the systemic threat human rights posed by the surveillance based business model underpinning the internet. Next we have Lindsay Kerr, who is a Democratic staff director and chief counsel on the Senate Rules Committee under ranking member Amy Klobuchar. She works on a broad portfolio of issues related to campaign finance, election law and national security. Next we have Morgan Williams, who is general counsel at the National Fair Housing Alliance, where he is responsible for leading the office's strategic and tactical legal initiatives and affairs, and where he directs NFHA's efforts to pursue pioneering litigation under the National Fair Housing Act. And last but not least, we have Natalie Marshall, who is a senior policy analyst at ranking digital rights where she leads RDR's policy engagement and policy development efforts. Natalie led the expansion of RDR's corporate accountability indexes methodology to address human rights risks associated with tech companies, business models, with a particular emphasis on the role of targeted advertising and algorithmic systems. Thanks for joining us. I wanted to first kick it off with a question for Joe. So in one of your latest reports titled surveillance giants, how the business model of Google and Facebook threatens human rights. You've outlined how control over our digital lives is one of the primary human rights challenges today, and it is something that we would never tolerate from a government. Can you talk a little bit about how we got here, especially with regard to the use of algorithms in targeted advertising. Thanks and thanks for the opportunity to speak with you today. So yeah, we looked at the business model that underpins the data economy, what some have called the original sin of the internet, which is, of course, the tech companies drive profit by knowing more and more about our digital lives in order to sell more and more finely targeted ads. And I think it's now widely recognized this model is broken. We, you know, analyzed it from a human rights perspective and we argue that this business is because it is predicated on surveillance ubiquitous and invasive surveillance of people's digital lives. It is a threat to human rights on a vast global scale, really affecting billions of people. And the rights that we're talking about here is, of course, privacy in so far as this business model is really the opposite of privacy. It relies on knowing more or less everything about you and when you operate in the digital world and using that to target you with with with ads. There's also this now knock on effects that have been widely documented on a whole range of other human rights, including non discrimination freedom expression, even freedom of thought. And this is all linked to the ways that tech companies have the power to kind of shape and control our information environment and influence our thoughts and behaviors. This is the use of the these this model on on their on their platforms. And we, I think we, you know, we've now seen these harms examples of the harms and the kind of knock on effects over and over again, you know, micro targeting of a political messaging in elections, you know, amplifying hate speech in some cases leaving to actual violence, enabling discriminatory targeting and illegally targeting children and collecting their personal information. This is all kind of comes back to the same problem with the ad driven business model underpinning these these platforms. We focused in particular on on Google and Facebook, not just as pioneers of the model but because of their dominance over not only digital advertising but over the the the global public square and the channel channels that we rely on in the modern world to engage with everything in the digital world in a search social media messaging video smartphones. And this goes hand in hand with the business model and it has the business model has both enabled these companies to to grow and obtain this this dominant position so it exacerbates and amplifies the the harms that we're seeing. So, you know, I think to a question actually just if I may really briefly on the question of kind of how we got there I think it's really important to look back at how this has evolved over the past two decades, because what's the point where we just accept this is the way the internet works, but we need to remember that firstly this the internet wasn't always rely on this business model and there are alternatives. And secondly that the problems that we're seeing now are entirely predictable. And almost two decades ago, the privacy campaign group epic spoke at the Senate warning of the long term implications of profile based advertising and the need for strong privacy protocols. And the fact that those warnings weren't heeded is why we are where we are now and that the self regulation has failed. And really we need to now look at how we kind of changing this this business model and putting a stronger oversight over the the the ad targeting algorithms, targeting delivery algorithms underpinning these platforms. Thanks Joe. Next I want to turn to Natalie. Natalie in your recent report series you talk significantly about the influence targeted advertising can and has had amplified through for example, Russian interference, and socialization and missing advertising. Talk about what specifically makes these algorithmic and systems so powerful, and why our focus needs to be on the business models of these internet platforms rather than just the content. Thanks Sandy and thanks so much for having me. So I want to start by by elaborating a bit on what Joe said about how the how the targeted advertising business model influences the way that the rest of the platform the part that makes users that makes users walk to the platforms, because I'm very sorry platforms but nobody's coming to your dear platforms for the purpose of finding targeted advertising that's just something that we kind of have to sit through to get to to post from our friends and family. And you know whatever else we want to engage with there. And deliver targeted advertising and to make a lot of money from it you need two things you need a lot of data about your users both who they are, and what they do. And to you need your users to spend as much time as possible so you have as much eyeball space, so to speak, to rent out or sell to your to your advertisers. So that creates a set of powerful incentives that have led the social platforms that that that we know and love or love to hate or hate to love, perhaps, and how we how we engage with them. So the comment so the, the, I think most of us by now are pretty familiar with the idea that engagement, meaning how much, how much eyeball time spent on how much you click how much you read is is is used as a product as the as the the metric that that company is used to to determine how much ad space they have to sell in a way and so everything about how the platforms work is designed to drive engagement. So that's why platforms want to show you content that they think you're more more likely to click on more likely to read more likely to have an emotional response to. And those are things that can be measured by computers. On the other hand, there's a whole lot of other things that I would argue are much more important. That can't be measured by computers. So how true how factually accurate something is how beneficial to society, something is whether something is a public service message message, perhaps about public health, or if something is, I don't know, sports score or the latest celebrity cheating right. A computer cannot tell the difference between these things that's something that requires human judgment. Because companies aspire to aspire to operate at massive scale, they have to rely on this kind of automation, which means that not only that the things that can be measured again by machines, end up being the things that count. Again, I would argue truly count, don't get measured and are at best after thoughts in, in the calculation of what content gets gets amplified and gets shown the most at worst gets dismissed as something that's entirely subjective and therefore not real and therefore not true. Right and that's where you get the discourse around companies not wanting to be the arbiter of truth. It's not because it's not possible to to to discern truth. In many cases, there there is a difference between actual facts and complete nonsense right. But it's not something that can be easily discerned by machines or something sometimes not not discernible by machines at all. The second bit and this this is where the data part comes in is that in order to to to to to in order to be in order to charge the highest rates you can as a company you need to be able to tell your advertisers that you have access to many more eyeballs and anybody else. So there again is where the scope and the reach comes in right and the need to rely on on automation. And you also want to be able to tell them that you have the best data to help them reach exactly who it is that they want to reach with their message. You know there's an old and old saying from from the Madison Avenue hey days of advertising that half of that all advertising money is wasted. The problem is that you don't know which half the the the promise of targeted advertising is to allow advertisers to know exactly which half is being wasted so that you're only showing detergent ads to people who actually buy detergent as opposed to the members of the household who have nothing to do with with detergent purchases that you only show your political ads to people who are registered voters in the correct jurisdiction and who are either persuadable or you know likely to turn out for your party but you just need to you need likely to vote for your party but you need to motivate them to actually take the the step of voting. So what's what's the what's the downside of all this well the downside of all that is that first of all much of this data and I would argue most of this data is collected in illegitimate ways, either because people don't know that the data is being collected that it's being collected in a coercive manner that you can only use the service that you need in order to participate in society by allowing this data to be collected about you, or it's then used for purposes that are not the purposes that you agreed to. So one example would be when you when you get your cell phone number to a platform to use for two factor authentication, which is a way of adding another measure of security to make sure that people who are not you can't hack into your account. And then that cell phone number is used for for your advertising profile or is sold to, you know, a host of shady actors who then use them to, to, to, to, you know, call you and try to sell you things or for for robocalls. That's that's that's a breach of the, maybe not a in the sense of breach of contract but certainly in the in the in the moral sense. And so because so much of this data is acquired in an illegitimate way that means that anything that you do with it is fruit of the poisonous tree to use to use a legal framing. So it so it's illegitimate but but then all in addition to that, and that I think should be reason enough in my view but as Joe was saying in addition to that it leads to discrimination harms and that's something that Morgan will talk a lot about I'm going to go into the next of his organization's lawsuit and the settlement but I won't so I won't spend too much time on that. But, but it also leads to violations to to freedom of speech to access to information and to to freedom of thought, as Joe said, so you know so I think we have to think about what the that not just the specific harms that come from the that they are real, and Morgan's going to talk about that. But we also need to consider the the impact the set of incentives that that creates and how that leads companies to create content curation algorithms the ones that decide which which of the many posts you could be seeing, you end up seeing and in what order, and how that in turn impacts the way that you perceive the world and the way that that that you operate within it. Thanks Natalie Lindsay I wanted to turn it to you now. Since the 2016 US presidential elections and which we saw foreign actors influencing the elections through the use of social media platforms and their targeted add systems. Senator Klobuchar who you work for has been a fierce proponent of expanding disclosure requirements for political ads online. Could you talk about the honest ads act which has been the main vehicle for these efforts, and how it encouraged greater transparency and accountability from platforms. Sure, thank you, Spandi and OTI and for all the other panelists for hosting me today and participating. As you mentioned, Senator Klobuchar is the top Democrat on the Senate Rules Committee which has jurisdiction with a little election. And she joined the committee not long after the 2016 election, but we know Russia so recently interfered in our democracy. And so the top priority for Senator Klobuchar since she joined the committee is to secure elections and to stop about the spread of disinformation online. And out of the gates, you know, the hearings of the hearings after 2016 about exactly what the Russians did. And one of the things I think how rich people so much was that we know that Russian operatives bought ads on Facebook with Google. And so we got to work very quickly after learning, learning that to find a way that we can build more transparency and accountability on online platforms, specifically related to online political ads. And one of the things that we came up with with the Onset Act, which is a very simple legislation that says if you sell political ads online, whether they be candidate ads where you're advocating for organesies, some of the election, or an issue ad, which is really important because we know that organesies after a huge issue ad to so to so just put in a vision a lot and they target those ads specifically. I think in 2016, the Senate intelligence committee said that African-Americans were the most targeted group in terms of disinformation for the adversaries. So the issue ad is a really critical piece of that. So if you're selling issue ads on your platform or political ads, those ads have to have a clear disclaimer from them. And you have to have a disclosure ad where the public, academics, journalists can go and easily access them. That is consistent with what the law that we have in the book now for ads that are sold on TV, radio, and print. And when those laws are written, the proliferation of online ads will go on. And so our technology has taken off in terms of the number of ads that are sold online, billions and billions of dollars, and our laws have not caught up. And so the honest ad that puts online political ads in the same thing as other ads is a really simple bill. It has bipartisan support. So when we first introduced it in 2017, right after the election, the late Senator McCain was our lead Republican on the bill, which made sense because he was, you know, the architect of McCain's fine bowl, McCain's fine bill, and he was very passionate about disclosure and disclaimers for campaign ads. And Senator Warner is our Democrat, Democratic partner. He's the Vice Chair of the Senate Intelligence Committee. So he understands probably more than anyone. The, what exactly for an adversary is you're doing how they're using platforms and then doing them to undermine democracy. And when we reintroduced again this year, Chairman Graham, who's the chairman of the Senate's fishery committee is the two Republican sponsor of the bill.  It seems like common sense. I think if you put it on the Senate Board today, it would ask, already asked the House, the Republican leader doesn't support the legislation. So that's our roadblock, but it's a really simple piece of legislation. And now with what we're seeing, just the sort of the manipulation and the craziness that we're seeing online. And I think a lot of people would say that the Honest Act is an absolute basic necessity and something that we should build off of in terms of transparent and accountability for online advertising. And I'll say, you know, when we introduced the legislation, we did work closely with some of the data platforms to see, you know, how it could work. And to Natalie's point, because a lot of ads, all the ads sales are done without human interaction, it is difficult. And the platform initially said, we don't think this is, you know, at least entirely something that we can implement. And then quickly realize that it could. And platforms like Facebook and Twitter have begun voluntarily implementing the Honest Act. So they do have disclaimers. There is an ad library. None of it is perfect. It's not, you know, complete, it's not complete implementation. And from Ranking Member Publishers' perspective, I think what you'll, what she would say is, you know, it's great that they're voluntarily doing this. It is absolutely not a substitute for passing the bill. And we cannot not trust these companies to stop this. I think that's really, really self-evident. So I'll stop there in terms of explaining what the bill does. Thanks Lindsay. And Morgan, I'd like to turn it to you next. So in March 2018, NFHA and three of its member organizations filed a lawsuit against Facebook alleging that the company's ad platform enables landlords and real estate brokers to exclude people of color, women, people with disabilities and other protected groups from receiving housing ads. The lawsuit resulted in a settlement that drove a number of changes across the company's ad platform. Could you talk about the scope of the lawsuit and what kinds of changes it resulted in? Sure. Thank you so much for the chance to be here. The National Fair Housing Alliance is a national office that works to ensure compliance with the Fair Housing Act. It's based in DC, but it has a network of local offices that are our membership across the country, local fair housing centers. And it's with three of these offices that we carried out an investigation of Facebook's ad platform and ultimately pursued federal litigation in this other district of New York. And in regards to the alleged discriminatory conduct of the platform's operations in the algorithms that manifest the targeted categories that were used in the ad targeting features, as well as other concerns associated with the operation and the ad and its delivery functions. In short, in actually in November of 2016, ProPublica ran an ad article featuring Facebook's ad platforms and the extent to which you can target those ads based on ethnic affinity. And essentially that was a term that Facebook advertisers used to refer to race and racial targeting of advertisements. And so there was a significant outcry as a function of this. We heard from our members across the country from our federal legislative partners in regards to concern over this and immediately reached out to Facebook. To get a letter noting concern over what was identified in the publication and what we had identified in some preliminary investigative work as a function of the outcry. Facebook was responsive and asserted that they would be changing their operations we've met with them several times in the following weeks. And that February of 2017, they noticed that they were they were changing their platform to remove these discriminatory features in their in their targeting operations. The following fall public were in a second story that showed that Facebook could not in fact change its operations. And the same as the ProPublica ran before they were able to to run and at that point we shifted our gear from sort of shifted our focus from advocacy with Facebook to enforcement oriented investigation under the Fair Housing Act. Very strong organizational standing or the ability for private housing offices to pursue investigations and enforcement of discriminatory conduct in the housing market. And Fair Housing Act is very clear that it's illegal to express the discriminatory preference in advertising and it's not only illegal for housing providers or housing service providers to do so, but for publishers themselves to publish those discriminatory ads. And so there's a long history of enforcement against publications. Nafa, in fact, in 2009 brought a lawsuit against American Classified, the largest classified company classified advertisement company in the country at the time for both print and online ads. And we were involved in amicus engagement in the 2009 litigation in both the Craigslist.com, 7th Circuit Decisions and the roommates.com 9th Circuit Decisions. The first set of 9th Circuit Decisions they went before the 9th Circuit in that case and dealing with the issues of the scope of the Communications Decency Act immunity to housing discrimination claims in in those cases and in subsequent cases. And we then and since have long adhered to the to the principle that the Fair Housing Act should not be relegated to print advertising and should apply online. Just as it does to print advertising and the immunity of the Communications Decency Act should not apply to housing discrimination. And in fact, in our litigation against Facebook in the motion to dismiss briefing that was filed, they asserted communications decency act defenses. And we briefed our responses to that. Notably, the Department of Justice filed a statement of interest or amicus brief and support of our arguments around the application of communications decency act immunity and application to Fair Housing Act Facebook operations. Because we settled that legal decision was not rendered on those open questions, but there is currently other briefing that's pending in other litigation in which Facebook is similarly asserted this defense in which there may be further case law on this question. In any case, we did reach a settlement with Facebook in which in partnership with a couple of other pieces of pending litigation. In involving the employment sector in particular, in which Facebook agreed to change their housing ad platform to preclude housing ads that are in regards to housing. Credit or employment services to not allow those advertising advertisements to engage Facebook's targeting ad platform in a number of notable ways. One, it drastically limits the targeting features to a set of targeting options that are jointly agreed upon by the plaintiffs and Facebook. Two, it restricts a lot of the targeting features around geography and other things that you could use to engage in discriminatory, segregating advertising practices. And three, it limits what was referred to as Facebook's lookalike feature to a now reconstituted special ad audience, which removes a lot of the filtering features on Facebook's ad platform tool that allows advertisers to target existing customer base or identified pool of individuals. We still have outstanding concerns about the reconstituted special ad audience to the extent that it's limited to targeting users on the basic basis of like internet usage. And we think that there's probably a lot of instances in which that's problematic. And under the settlement, Facebook specifically agreed to study that operation and the discriminatory effects of that operation and confer with us about further changes to that part of the platform. It's no worthy. And I'll stop here. Definitely ramble on further about this and really welcome any questions in particular coming out from listening to the roundtable discussion that preceded this is very compelling and some no early, I think relevance to some of the work in this case, but would just add that there's a HUD investigation that was reopened in part as a function of the lawsuit that we have filed. But it was a secretary initiated complaint that HUD issued a charge on in March of 2019 shortly after we announced the settlement of our of our lawsuit, which referred the matter to the Department of Justice for enforcement. And then considering kind of outstanding concerns around Facebook's operation, I would say that they're principally twofold. One is in regards to the potentially discriminatory function of their special ad audience. Feature has been reconstituted and two is in some of the research that's been published more recently since our settlement on the delivery function of Facebook's ad platform. And the extent to which separate apart from the ad campaign that is programmed into what is launched into Facebook's ad platform that the algorithm that Facebook uses to sort how those different campaigns are prioritized and delivered to viewers. Excuse potentially skews results in a discriminatory way and there's outstanding concern about that. The Department of Justice is not filed suit. So, ostensibly, it's likely that they're in some form of settlement negotiations. And it may be that they're focused on these two outstanding issues that are subject to concern. Thanks, Morgan. So now I would like to open some questions up for the panel as a whole. So as we've discussed the use of algorithms for ad targeting and delivery purposes can perpetuate harmful outcomes biases and discrimination. What kind of mechanisms can be leveraged to reduce the instances of such harmful outcomes and promote greater transparency and accountability around the instances that do take place. And some examples that I know we've all discussed and mentioned in some of our reports are algorithmic audits and impact assessments. So I'd love to get your thoughts on that and maybe first I'll turn it over to Natalie. Thanks, Bandy. So I think there's there's three main areas of intervention that that I think are particularly fruitful for the US Congress to pursue. The first one is mandatory transparency that goes that, you know, I think cannot can and probably should start with the honest ads act, but going beyond that to apply to not just political ads but other types of ads in general. And because I think I think if among the many, many lessons we've learned over the past six months of life in the time of coronavirus is that political ads is not the only place that harmful disinformation can can circulate right. So that's that's the first place, but also looking at transparency around how all content both paid content like advertising and user generated content is governed on the internet. This is very different from having the government set the rules. Right. We're not that is that is not an appropriate thing for the US government to do that in fact would be contrary to the to the First Amendment. But I think it is appropriate for the government to require certain measures of transparency about what the rules are how they're enforced.  Right. That's not what we should aspire to we should aspire to having a functional process and then publish numbers about about the outcomes, how much content is taken down what type of content through what mechanism, etc. The second place where I think and that is think is going to be the most, possibly both the most difficult to negotiate but also the most impactful is federal privacy legislation. You know there's been a lot of talk recently in Washington is likely to continue to be about reforming or possibly even eliminating the communications decency act. I'm very sympathetic and agree with most of the goals that are being pursued through this avenue but I just don't think that the that intermediary liability is the way to get to that. Perhaps clarifying, you know that what was and was not the original intent of the CDA because it there is a growing legal consensus that many of the court, the court decisions that Morgan referred to were decided in ways that were just not consistent with what the CDA was actually meant to do. I think that that's something that's that's worth talking about but I think more importantly than that the way to prevent these harms from happening is through privacy legislation. We have a framework in in our latest report series. I'm not sure whether we're able to share the link through through a chat here within zoom or not, but if not I'll certainly be tweeting it out. That outlines what we think a federal privacy legislation should should should focus on so that the types of harmful discrimination and illegitimate data collection that that we've been talking about can't take place. And then the third piece, which I think is probably the wonkiest and perhaps least sexy to talk about has to do with corporate governance reform. It has to do with how companies govern themselves. There's been a lot of talk about how the CEOs of Facebook and Google in particular have these dual roles of being both CEO of the company and chairman of the board of directors and on top of that, you know, a dual class share structure that that's that's very popular in the tech sector. They have more, they control more more votes than anybody else which puts them in a position of essentially being dictators of their companies and are completely immune from the type of oversight that a board of directors, or the fact that they have shareholders right in our publicly traded company is meant to do. So I think there's there's a lot of innovation to have there. There are actions that entities that are not the US Congress should be pursuing. You know, I think antitrust is and I know that there's a there's been a house judiciary hearing just announced for later this month that we'll focus on but that's, I think that I think at this point is mostly out of the hands of Congress and in the hands of antitrust regulators. So I think I think those are the, but it is it is indeed worth pursuing, just on a separate track from the legislative options that I just mentioned. Thanks Natalie. And would any of our other panelists like to jump in. Yeah. Let's go with Joe first. Okay, thanks. I'll just be very brief just, you know, certainly we welcome all of the recommendations that Natalie has just outlined just building on that for a couple of others from our point of view. I mean, I think going back to this, this, this point around platform power and the the dominance of a handful of tech platforms. I think, really, we need measures that will, that will tackle that and encourage a more kind of pluralistic Internet. So, antitrust could potentially be one of the tools in the arsenal of states and regulators and interoperability as well I think is an important technical standard which should be which should be much more greatly enforced to enable others other platforms to develop and then I think one other thing which we think is really high priority is really to make it lead new legally binding protections to stop platforms from forcing users to accept this surveillance business model because currently how what we see is that the effectively users have no choice but to sign up to this kind of invasive profiling and targeting in order to use services which which everybody now relies on. So really you have a false choice where either you don't benefit from the modern world and the digital world, or you consent to this model which is abusive of human rights. And so I think that kind of needs to be a first step the government should look at in so far as people should have the choice to opt out meaningfully from these kind of practices. So, and Morgan just before it over to I just want to remind everyone in the audience that if you would like to ask the question to use q amp a function. And now Morgan onto you. Thanks just a brief comment in regards to the question about tools for accountability and transparency in terms of civil rights enforcement and accountability for civil rights protections one of the most important tools that we have at our disposal is disparate impact liability which is separate apart from intentional discrimination claims but are involved claims in which a policy that's neutral or non discriminatory in its on its face when put into practice has a discriminatory outcome. And that can be a very powerful tool and talking about algorithmic operations. Unfortunately, well that's a very powerful tool that's at our disposal HUD in 2013 issue the rule that provides a lot of common understanding for what a standard of disparate impact liability should be unifying a lot of the different circuit analysis that is out there. In 2015, the US Supreme Court issued a decision in Texas Department of Community Affairs Housing Community Affairs versus the inclusive communities project that upheld disparate impact liability under the Fair Housing Act. And it's actually on the basis of this Supreme Court decision that the current administration is actually proposing a new disparate impact rule. That would completely upend the ability to bring different impact cases and in particular has a specific defense regarding any practices that are associated with algorithmic operations in which the inputs used in the algorithmic model are not themselves. Substitutes are close proxies for protected characteristics and the model is predictive of risk or some other objectives and anyone who knows algorithmic models knows that that's an absurd defense because it's not individual inputs that contribute to the output of an algorithmic model but the relationship between different inputs and so such a defense. Really would provide a kind of safe harbor immunity for any practices that are based on algorithms. This is a proposed rule that HUD accepted comments on this past fall and a number of folks in the tech civil rights world. We're very responsive to and really appreciate everyone's engagement on that in partnership with the the Fair Housing and broader civil rights world. HUD sent a proposed final rule to OIRA, the agency responsible for securing final interagency review of that final rule in early May, May 7th, and is poised to issue that final rule any day now. So please be on the lookout. I think if you are interested in engaging in advocacy on this issue, there's the opportunity to actually schedule meetings with OIRA, the agency that's coordinating that interagency review and confer with them about your concerns about at least what was in the proposed rule. We don't know what's in the proposed final rule now. Additionally, we'll be looking to challenge that final rule on the basis of potential Administrative Procedure Act claims and there may be avenues for some folks in this space to be potential litigants and challenging provisions specifically that deal with algorithms. Additionally, we'll be working with any future Administrations to rescind these terrible new standards that are being put out there to roll back to the rights protections in this space. Thanks, Morgan. Lindsay, would you like to jump in? I'm sorry, Lindsay, would you mind coming a little closer to your screen? It's hard to hear you. Sure, is that better? It's a little better, yes. Thank you. You know, I think one of the things that Natalie said she talked about Section 230 reform, and that's something that is being discussed heavily in the House of Congress right now. I think that there are multiple Senators on both sides of the aisle who are very interested in seeing sort of different bipartisan bills crop up. Many conservatives are motivated by the myth that the algorithms are biased against them. And I think that on the Democratic side, we're motivated for all the reasons that we're discussing here right now. And, you know, I think that's something you're going to see more of. I know Vice President Biden has made clear he supports the reform and many others. I also say, you know, I think when we have regulators like the FTC issuing fines for long doing and lawlessness on behalf of some of the platforms this year ago, we had the FTC decision about Cambridge Analytica and the $5 billion fine, which we could find ever, but, you know, when you make $55 billion a year, it's nothing. So I think having, you know, having regulators really, you know, to compete behind some of the fines for the person is a long way as well. Thanks. Turn it now to audience questions. The first one, what should multi-stakeholder collaborations around promoting accountability around digital advertising algorithms look like? In particular, how can industry, government, civil society and civil rights groups that are engaged, and what are like the main challenges that you see in trying to push this kind of work forward? Maybe I can turn it to Joe first, if you want to take that. Yeah, sure. I mean, I think that there is a lot that can be done in the short term through kind of multi-stakeholder collaborations. I mean, I think following the previous question, ultimately this is going to require state-based regulation and binding laws, self-regulation has failed. But I think at least in the short term there are discussions that can be had between affected kind of communities or representatives affected communities and the big platforms in order to put in place better kind of systems and mitigation measures that the companies can adopt. I mean, actually, I think that the Stop 8 for Profit campaign, which is obviously very live at the moment, you know, I think that's a good example of, you know, there's a lot of issues there which overlap with what we're discussing and which really tie back to the problems of the business model. And there's some really concrete recommendations that the campaign has, like putting in place a C-suite executive with civil rights expertise in Facebook. And I think those kind of steps would be really important and really impactful until we can put in place these longer-term laws and regulations that actually protect people kind of across the board. Thanks, Joe. Yeah, so I think, you know, I think there's been an evolution of the civil society and multi-stakeholder discourse around platform accountability. You know, I think the early years of social media platforms in general were filled with a lot of optimism around the positives that these new tools could bring for activism, for social movements and more. And, you know, starting around, you know, the middle of the past decade, I think there was a growing realization of the harms that could come to movements and to activists and others from relying so much on these platforms and what happens when, you know, an activist content gets taken down and how can you get it restored and how can you appeal and so on. And then I think after, you know, I think the year 2016 was a big wake-up call certainly for social movements in the West, you know, with the twin tragedies of Brexit and the 2016 US election. And yes, I do view both of those events as tragedies. And because they were just such clear self-owns by two of the world's most powerful democracies. But I think since then we've all collectively spent a lot of time reflecting and analyzing and looking at data to better understand exactly what happened. And I think that that understanding exists there now. It's disputed to be sure, but I think we do have a solid collective understanding of what happened. And I think now we're – I think now that what's happening is the building of a consensus around what needs to be done. And I've been – I am encouraged by the progress that's been happening over the past few years.  And now the debate that I see happening is on exactly what the tactics are for getting there. And I do think there's a shared vision for what the outcome needs to be, right? A technology sector that is more responsive to democratic oversight, to human rights, to the needs and interests of the public and not just to their own bottom lines and their shareholders' pockets. But now we're talking about what tactics will be needed to get there. And I for one am hopeful that that's a conversation that's going to mature very quickly, especially if new channels for activism emerge after the 2020 election here in the U.S. Thanks, Natalie. Morgan or Lindsay, would you like to jump in? Okay. So the next question we have is for Morgan. How do you think lawsuits, such as the one that NFHA helped lead, can play a role in promoting greater accountability around the use of these algorithmic systems? And what are some of the key lessons that were learned in the process? Yeah. Well, I think lawsuits can play a very important role in changing the practice of individual operators, but also in informing the market more broadly about where prospective liability may lie. Just recently announced some changes that Google was making to its platform in regards to its targeting features in the housing space and in particular around the use of zip code and other geographic targeting features as they may be used in the housing sector. And that kind of enforcement can help to educate the market about the scope of that kind of liability. Additionally, there are open questions about the scope of the relationship between the Federal Fair Housing Act and the Communications Decency Act. And those questions may be decided in some fashion through legislative means down the road, but there are open questions that may be decided in the courts in future litigation. And though our case settled and those legal questions were left open, there are other cases that remain and other litigation that we may pursue against other parties that may help to sort of clarify some of those legal questions in ways that we think would expand the scope of civil rights protections in this space. Thanks, Morgan. The next question is for the panel as a whole. So we've talked about forms of legislation such as the Honest Act, which is an example of policymaker action that tries to create a transparency around online advertising. There have also been other bills that have been introduced and as Natalie mentioned, some of them have raised some First Amendment concerns because they tried to direct how platforms regulate their content. So I would love to get everyone's thoughts on what should policymaker, sorry, what should policymaker action look like in this space. Do you think that sort of the bills that have been put forth go far enough, do you think they need to go farther? And maybe first I can either turn it over to Lindsay or Natalie. I think the very clear set was that we talked about it, that this legislation is not, it just started to work. But like I said, I really think that there is, you know, more must be done. And right now. Thanks, Lindsay. Natalie? Yeah, I mean, so I'll just briefly repeat what I said earlier, which is that, you know, I think first we need, you know, we need transparency about how content is governed online, both with paid and unpaid content. And the Honest Ads Act is like Lindsay said, a no-brainer place to start. You know, it's unfortunate that there's a one-man obstacle in the way of getting that bill passed. It is what it is, at least for now. The second thing, which I think, again, the most impactful thing by far will be to pass strong privacy legislation at the federal level. And I think one of the goals of that should be to make the types of discriminatory targeting and the disparate impact that Morgan talked about be as close to impossible to enact as can be done. And here again, I'll point people to the Ranking Digital Rights Report, which I think the OTI account graciously tweeted out the link to for details on that. And then the third bit is reforming corporate governance so that companies themselves have to be accountable, not only to Democratic oversight, but also to their boards of directors and to their shareholders. I think once those things are in place, I think at that point it will make it will make more sense to look at at section 230. There's some interesting proposals out there. So, you know, I mentioned, you know, clarifying what the original intent was, you know, and certainly clarifying that disparate impact liability laws, you know, that the CDA should not be a shield from that. That's something that I think there's very sound legal arguments for public knowledge also has a has a very interesting proposal out to carve out advertising specifically from CDA protection, and I think that's that's worth discussing. But again, I don't, I think it's going to be much more impactful to focus on transparency privacy and incorporate governance reform first, especially since those three avenues don't have the risk to freedom of expression that intermediary liability does. Thanks, Emily. We're nearing the end of our time but I want to give Morgan and Joe a chance to jump in if you have anything to add. I'll just add one point really briefly. I mean, I absolutely agree with said by Natalie Lindsay just now on these points. I think ultimately, the, there needs to be oversight and accountability. I think that means that companies are accountable for harms produced by the optimization decisions made by their algorithmic systems, whether that's targeting at delivery or the engagement and kind of personalization algorithms on the platforms and I think transparency. We need transparency in order to be able to know what those outcomes are. And then, but, but fundamentally, the only way that we're really going to shift this is is for the companies to be held legally accountable when when they, they're a human rights harms directly linked to the operations of their system. Great. Thanks Joe. So we're now ending nearing the end of our time. Thank you so much to everyone for joining today's events. Thank you to our wonderful panelists for taking the time to speak with us today. I hope you found this conversation productive and helpful and beyond the lookout for future work that I'm sure all of us will be doing in this space. So that will be on YouTube if you'd like to access a recording afterwards. And yeah, thank you and enjoy your day.


I think we're good. Is this? Yes. Okay. We're gonna move this. Yes. Hello. Hello everyone. Thank you for coming to Metcalf Small. My name is Nandan. I'm here to introduce our speakers, Alex Hovind and Landon LaSmith. They're gonna be talking about scaling your open data hub for fun and production. Please welcome them. Thank you. Thank you. Do you want to share your places? Uh... I'll drive for you. Test, test, test. Hi. So I'm Landon LaSmith. I am on the team that's primarily responsible for working on the open data hub. I'm passionate about Linux and all things containers. And unlike Alex, I am not a beekeeper, but I like honey. Alex is much more interesting and I'll let him introduce himself. Cool. Yeah, so I'm Alex Corvin. Like Landon, I'm a software engineer here at Red Hat. I work sort of on the open data hub, but I'm primarily focused on running Red Hat's internal instance of the open data hub. I'm kind of like a systems engineering devops, SREs, I reliability engineering kind of guy. That's what gets me excited. So like scaling systems, that sort of thing. And yeah, unlike Landon, as he said, I am a beekeeper. So, you know, if you want to talk about bees after this, yes? Do you like honey? I don't really like honey. It's weird. I like having projects and I get bored if I don't have something to keep me busy. So, I saw a post on Reddit one day about beekeeping and I thought it seemed interesting, so now I'm a beekeeper. But yeah, that's who I am. So what you can expect from this talk is we'll just introduce the open data hub. And we'll talk about fun versus production with the heavy emphasis on production, which is Alex's expertise. Challenges that we faced while scaling our internal instance of the data hub and then answer any questions. So what is the open data hub? So first and foremost, it's a reference architecture for running data and machine learning projects on OpenShift. So we want to develop a community where we can give you best practices for how to deploy different tools on OpenShift environments in OpenShift environments. And our primary way of doing that is development of a meta operator. So a meta operator is essentially we are an operator that will handle the deployment of other operators and custom resources in OpenShift. We're not trying to control the whole stack. We just want to make it easier for DevOps, data scientists, data engineers to deploy an environment on OpenShift. So data hubs. So at a high level, these are kind of the core parts that we're focusing on. So in the DevOps, we have support for monitoring, optimization, and model serving. The data engineer, we primarily focus on data storage using self-object storage. So you can use your favorite kind of S3 commands, AWS, CLI, to push that data and access it. And right now, the Open Data Hub is focused on making it easier for data scientists to perform their workflow on OpenShift. So the Open Data Hub is primarily focused on the yellow and green parts for right now. Alex is working on the internal data hub whose core focus has been on the blue and green with some intersection for the data scientists. So this is a very busy diagram of the low level focus of the Open Data Hub. At the top, you'll see we have different tools for AI machine learning. Right now, we are focusing on Jupyter Hub to allow the data scientists to interact with different tools. We have the Open Data Hub and AI library for ML applications. AI library is a collection of different models that you can import or use to, I guess, model your data. Selden is currently in the Open Data Hub so you can create models and a few other things. Alex right now is working on the data analysis portion, metadata management, and I think storage also. So as the Open Data Hub grows, we are working to bring a lot of those internal data hub features that we've been using in production into the Open Data Hub. So the Open Data Hub is fun. This is as little as it can get. We want to make it easy to allow you to deploy different tools into your environment. So it's modular, so if you want Jupyter Hub and Spark, but you don't care about monitoring, you can do that. I think we have an instance of it running internally where we're only focused on the Spark operator for deployment without Jupyter Hub, but it's completely up to the user when they deploy it. So we want to make it easy to simplify the deployment and redeployment of your tools. So if you want to play around with different tools or data sets, you can do that.  So that's the fun part.  How do we plan for production? So this will just be kind of like a quick talk. Alex has all the good information about how they actually do it in production, but with the Open Data Hub, we enforce modularity. So if one product breaks, we don't want it to bring down the whole stack. That's one of kind of our core tenets for adding new components to the Open Data Hub. So a component is something, the individual piece that we're deploying. That could be Ceph Object Store, Kafka, Prometheus Grafana, Jupyter Hub, Spark operator, and anything that we add in the future. Reproducible behavior, so we want to make sure that if this does grow into a production system, that you can easily redeploy it with the minimum amount of effort and have everything work. And the key things we're trying to focus on to make it ready for production is to get as many use cases as possible, because we want to know what breaks we can make the Open Data Hub better. That's probably the most important thing that we're asking for for production. So the Open Data Hub is a community. So we want as many people to use it, many people to contribute new components so that we can figure out the best way to make it better. All right, so thank you, Landon, for giving the intro to the Open Data Hub. I'm going to real quick give you an intro to what do I do on the internal data hub? What does my team do on the internal data hub? Can you guys hear me okay? I feel like I'm not doing a very good job of speaking into this, so let me know if I start breaking out. So first and foremost, the internal data hub, I mean internal at Red Hat, we just call this the data hub, right, is a platform for enabling teams at Red Hat and environment for enabling teams at Red Hat to do what we call become data centric, right? We want teams to have an environment where they can store their data and where they can reliably get access to that data and where they can explore and implement actual production workloads of various data science tools, right? So we maintain this environment, we make sure that it's stable and reliable and meets their needs. And then a really big part of what we do is reaching out an enablement to these teams to make sure that they know how to use our systems so that they know how to use the components of the open data hub, so Jupyter Hub, Spark, what have you, and guide them through and tailor like our walkthroughs, our tutorials, our guides for actual meaningful use cases for teams at Red Hat, right? So that's one really big part of our charter. The other thing that we do is we're kind of a proving ground for the open data hub itself. Kind of the model we've been working with is if there's a new component that we want to add to the open data hub reference architecture, usually we'll run it internally first and make sure that we understand how to run this thing and we can do it so reliably. We kind of work out all the kinks before recommending to the world that they run this in their environments, right? So we're kind of the guinea pig, if you will, for some of these components of the data hub. And I'll get more in, I think, the next slide, actually, about some of those things we're working on right now. So we're a proving ground, but also related but an important facet dimension is we are kind of the customer zero initial point where all these components get to run at scale. So we're not crazy scale right now, but we're solid scale for you 100 gigs of data running through our system a day. It gives us an environment with which to really prove out, again, that these components can actually work in a production environment. So I mentioned that, you know, we, you know, new components to the open data hub typically go through us first. I wanted to touch on a few different things we're working on right now. The first is Kafka. Malakir back in the room has kind of been owning that. That, you know, will actually will be a big part of my talk later on. But we wanted to, you know, prior to Kafka in the open data hub, our processes did not really scale. We didn't really build like a very effective streaming data like platform, right? It was really hard to scale our processes and scale the architecture and scale how teams sent data to us and got access to it, right? Kafka has been really helpful there and it's like a very strong recommendation that will make to users of the open data hub when building out their data like platforms, right? Is leverage Kafka for that? And again, I'll get more into that. Anyways though, Kafka, I think it's in master now, right? Like you can deploy Kafka with the open data hub operator. I think that will be officially released and out on like Quay for you to pool in a couple weeks, right? End of the month. So the next thing that we're working on, we call this kind of overall like we lump them all together in what we're calling the data catalog. Specifically though, there's three components. One is Hive. I'm sorry. One is Hue. One is the Hive Metastore and one is the Spark Thrift Server. So together, these do a couple kind of cool key things. One is they give you a kind of a a pane or a window through which to query your data and explore your data and move your data using familiar tools specifically SQL. So your data scientists, data analysts who are familiar with executing SQL queries can do so on your data now that's stored in like your S3 data like using traditional SQL tools. The other thing that it lets you do is build out like rich sets of metadata about all of the different data sets that you have and store like sample data and schema data and you know arbitrary catalog information that you want about like who owns the data or what is the SLA for this data, what is the volume of data, et cetera. Store that in a database kind of alongside your data and then get really easy access to it through Hue. So the Thrift Server provides the ability to execute SQL, the Hive Metastore execute provides the ability to store metadata and Hue provides the ability to explore all of that. So these are three components that we've been leveraging now for a little while to build out abstract transform load, I think. Pipelines internally here at Red Hat to get our data into a place where the data scientists can then work on it. Kind of working out the kinks, it's working pretty well now though internally and that's another thing that we're hoping to have kind of out there publicly available for use at the end of the month. So finally, the third thing we've been working on now for a while internally is Elastic Search. We'll talk more about Elastic Search later on but we have a lot of teams internally at Red Hat who run their systems in production, right, typically on OpenShift and these systems generate a lot of log data and, you know, if a production issue happens you want to be able to get access to these logs to see what happens. The problem is OpenShift pods are ephemeral and if the pod logs are only stored with the pod it can become really difficult to get access to those logs because the pod is not there anymore. So we implemented Elastic Search, this has been around for a little while but we've kind of recently wrangled the beast if you will and are still working on that, but it provides a really solid platform, Elastic Search with Kibana to visualize them for groups to be able to send us their operational logs and view them in pretty much real time and see exactly what's going on and find like if an event is happening across multiple pods or whatever, there's a lot of powerful stuff with it. So Elastic Search, I don't think is officially on the roadmap right now for adding to the OpenData Hub operator but I think it's something we would like to do in the future. But so that's something we work on a lot internally is making sure that that's ready for prime time and that could be like easily installed. As Landon said, we want to make sure that components of the OpenData Hub are really simple to get going with so we work on that a lot. We want to get Elastic Search to that kind of place before we'll officially make a part of it. So anyways, the full roadmap to the OpenData Hub operator is there if you want to check it out and see exactly what we're doing and when we plan to have it done. So enough of that, now we're getting kind of like the exciting stuff, right? I wanted to talk about some challenges that we've faced internally while scaling the OpenData Hub. My hope is that this will not be too specific to the OpenData Hub that like if you don't plan to run the OpenData Hub you'll still get some value from it. Some of it might be specific to running like a large big data platform or a data lake but hopefully you'll be able to get some kind of generic nuggets out of this to use in your own production systems. And there are I believe four specific areas that I want to talk about, four specific challenges and how we're kind of tackling them. So the first one is monitoring. If you came to my last talk you know that this is something I feel passionate about that like you should not feel comfortable calling a production service like production ready if it's not monitored and if you don't have alerts in place and if your team does not like understand what to do with those alerts, right? So this is something that we've spent a lot of time on and one of the cool things about the OpenData Hub operator is you get Prometheus and Grafana monitoring for free and so like you know my team's job is to make sure that the internal data hub is like stable and reliable and available and a big part of how we do that is monitoring. We typically do that with Prometheus and then Prometheus is alert manager component which lets you pool metrics and then generate alerts based on those metrics and those alerts to however you want to get alerts, page or duty, email, whatever Slack. So first thing I'll say is like if you're running an application on Kubernetes or OpenShift, highly recommend that you play with Prometheus it integrates natively with Kubernetes and makes it really easier to pool metrics from your systems it's really easy to add custom metrics to your application if you have a custom Python app or API or whatever you run something more standard like maybe you run a database server or a patchy web server or something that's common. There are a multitude of public open libraries and they're called exporters in the Prometheus language but that will give you a lot of really rich metrics from your application without having to spend time becoming a super expert on whatever that application is to know exactly how to monitor, right? So that's the first thing is leverage Prometheus to be able to monitor your applications. So the other thing that I wanted to say that has taken us time to learn is just making sure that the team knows what the metrics and resources and alerts are rather. I think in the olden days we would deploy a service to production monitoring, maybe not a lot of monitoring and not necessarily taking the time to really understand the application and know what I actually need to monitor. This has been a learning experience for us with Spark and Jupyter Hub recently. We've had Spark and Jupyter up-deployed for a little while internally but it was kind of a wild west. I almost hesitate to call it a production service. We didn't really know what to monitor about it, how to effectively monitor, what was starting on what to do when alerts generated. So that's really the second big piece of advice I have for you here is if you're going to run something in production take the time to understand how to run it. If it's a database you can leverage for example the MySQL Explorer for Prometheus so you don't have to do all the monitoring yourself but take the time to research what do these metrics mean what do I need to care about and what's just kind of white noise, right? So that's what I have to say about monitoring and certainly I'm passionate about monitoring so if anybody wants to come up and talk to me more about it afterwards feel free. So the second area and this area I hopefully will have maybe some more nuggets, right? It's something that like my team knows we've worked really hard on and this is about scaling our processes so when we first deployed the open data hub internally, the data hub we were kind of in user acquisition mode, right? So the important thing was we get it running we get as many useful services running in it and then we work really hard to get teams at Red Hat using it and we were successful with that and I think a lot of teams might be, this might be a familiar story like you have success you get a lot of people running using your system and then more and more people will start coming to you trying to use your system and the problem is until now you've been so focused on doing whatever it takes to get people using your system it turns out you don't have good repeatable scalable processes for onboarding new users so for us we have a couple different storage platforms one is our CFS3 data lake one is Elasticsearch they have different authentication methods they have different patterns for writing data they have reading data they have different processes for adding a new user they're just very different and that made onboarding new users really difficult in Elasticsearch's example we had to add new certificates to the Elasticsearch system with read and write permissions to whatever sets of indices then we had to do a production rollout of Elasticsearch which took literally two days to do to add the new user and then for S3 there was this whole other process where you've got to get credentials you've got to assign users to buckets you've got to keep track of all that stuff it was very hard and ultimately what happened was a request for a new onboarding thing would come in and we'd see it and we'd say oh man that process is a pain to do I don't feel like doing it I'm just going to let that sit for a little while maybe somebody else on the team will do it and then two weeks later that request has been sitting there forever we better take care of that team morale suffers nobody wants to be doing this stuff it's not fun you're spending all your time doing onboarding stuff it takes a long time so now your team looks bad the important maintenance test don't get addressed so an example there is I'll get more in depth in details on this in a second but at some point in the history of our system we started implementing controls for how long data would be kept around in our system the problem was if we didn't have our processes well defined those life cycle policies wouldn't get applied so now the system would start growing kind of unfettered and would get just in worse and worse health over time because we didn't have systems to adhere to similarly our scaling considerations wouldn't get properly applied if there was a unique data set or a particularly large data set we would have to do a particular process to onboard them again without well defined processes that wouldn't happen without well defined processes we wouldn't know what data was stored in our system or importantly if there's personal identifiable information in that are there sensitive data sets we just have all these things that when we didn't have well defined processes for onboarding new systems a lot of stuff went by the wayside over time it just became harder and harder to manage the advice here is at some point you have to figure that out at some point you have to sit back and do the work and maybe it's painful but you have to do the work to figure out what the processes are get together as a team write down everything you have to do make sure everybody understands the process make sure it's documented make sure everybody on the team can do the process be really really explicit with what the steps are maybe you realize that like wow this process is really bad and this process is really hard and it does not work but you have it documented and then you can start to improve it and then maybe you can start to automate it that's kind of the phase that we're at internally right now it's been a little hard to get to but I think we're at a good place now instead of new user onboarding taking weeks it takes a couple of hours maybe and that's been really good as a result we're going to be able to go out and tell them this is how you should do it and that's kind of our goal scaling our processes that's been a big challenge the next big challenge that we've been working to overcome is sort of streamlining our architecture so I'm going to talk through this diagram a little bit but let me first say that if you're running a production data lake as your data lake grows and as you have more and more data sets going into it in our experience the logic for where that data goes or what to do with that data can become more and more complex in our case in our system we can store data in Elasticsearch or we can store data in our CFS3 data lake traditionally Elasticsearch has been used for things like log streaming and log analysis and CFS3 has been used for more data science data analysis kind of tools we're using for that traditionally a lot of the tools we use are really good at working with AWS and so CFS3 provides an S3 API that these tools can work with if a team wants to do data science we push them towards S3 if they want to do log streaming we push them towards Elasticsearch but there are cases where the team wants to do both long term we'd like to do something better but right now we just write it to both so there's duplication that's okay sometimes there are transformations that have to happen on this data before they get sent to their final resting place so there are different processes kind of different layers of processes that handle that and again going back to before we had this streamlined architecture it seemed like every time we added a new data source we just added a new component to the data hub that would do that new thing and it became very unwieldy and very a lot of duplicated work and just like hard to maintain that's kind of a theme here is it became hard to maintain so what we did is we introduced Kafka so Dashbox, Kafka, Strimsy Kafka Connect I think there have been other talks here maybe specifically Strimsy Operator if you want to run Kafka in Kubernetes or OpenShift and you don't do it with Kafka or if you don't do it with Strimsy use Strimsy it's the way to do it Strimsy is an operator that runs Kafka in OpenShift or Kubernetes so anyways we run Kafka in front of the data hub and all new writes, all writes of data go through Kafka and then it lets us do whatever we want so like if the logic is pretty simple about where it goes to it goes to Elasticsearch, it goes to S3 like we have very generic ingestion layers that will handle that so the data producers just basically like write it to the right place in Kafka with the right key our system handles where it goes if it's more complex we can have a very consistent layer of we call them normalizers to do that those transformations that are necessary and so they just read from Kafka right back to the right place it ends up in Elasticsearch or S3 right and so the benefit to this is that we don't have to worry about all the different like complexities of where data ends up we can do things in a very like generic way that will scale and then we just have to worry about the stuff that's like from Kafka back all of our data producers if they want to start sending us data we can create a topic for them in Kafka we can tell them where to write it where the writing is on them so and the other nice thing is that Kafka Kafka is very easy to scale you can add topics you can add topic partitions you can add consumers and that's maybe not the case with some of the backend things like Elasticsearch or S3 so if we have a burst of data we can scale up Kafka or messages can kind of queue up in Kafka and it gives us a lot of resiliency that we may not have on the backend so recommendation here I'm a big fan of Kafka if you're not familiar with Kafka as an enterprise message brokers with data streaming system like if you have large amounts of data if you have complex logic for where data should go or if you have complex logic for transformation of data highly recommend you use Kafka and so again back to the open data hub we spend a lot of time to kind of understand how to run Kafka and how to architect Kafka as part of a data streaming platform and we're really excited to be adding it to the open data hub operator and be able to provide it to the world to run as part of their data lakes on open shift okay so the next thing I want to talk about is managing the volume of data how am I doing on time? what do I have? 15 more minutes? cool so for this I put if you build it they will come I don't know if it's a baseball movie they'll watch it it's a good movie better than the rest of us talked so for us this has been really true I mentioned that for a long time we were working really hard on getting the data hub to be a stable platform where people can write their data to and I think we were successful with that and now we're struggling with our own success how do we make the data hub we're operating at scale now so that just brings all sorts of problems or challenges rather I should say if you build a shared data platform people are going to want to send their data to it and so one of the problems we've had is I think people have a way of sending you data and then kind of forgetting about it or you don't really know what data you have or that data is only useful for a period of time when you keep it around forever so three specific things I'll say here one is introduce controls for limiting how long your data is kept so introduce data retention policies this is really easy to do with Elastic Search there's a tool called Curator you can use to expire old indices it's really easy to do with CFS3 you can implement data life cycle policies utilize both and honestly for us we're not quite here yet it's not a strict default but we recommend have default policies for how long data will be kept in your system and adhere to them and require a good reason not to adhere to them for us we like to default to keeping data around for 30 days we can be flexible on that but it forces people to think about do I really need this data around and honestly sometimes there are use cases where there are regulatory requirements that you don't keep that data around forever requiring people to think about how long they store that makes them think it's better for everybody you spend less on storage it's easier to scale because you don't have to scale as much it's just easier so strict data retention policies is the first thing the second thing I'll say that has been really important for us in managing the volume of data is implementing a way to keep track of the data that's in your system so at some point if you have a bunch of people sending data to you it's impossible without a system to know what data do you have the problem like if you don't know what data you have then like why are you storing the data you want your data scientists or data analysts to be able to explore what data is available and be able to do interesting things on it you don't want 50 teams at your company to all be storing the same set of data like there's so many benefits to being able to know what data you have in your system and when you're working with hundreds of gigs or more of data a day you can't keep track of that manually so this is where the hue and high of meta store and spark thrift server solution that I mentioned earlier comes in we think it's going to be a really scalable mechanism for keeping track of all the data and allowing people to easily explore it so if you're running a data lake have a tool like that and then the last thing here that I want to talk about is keeping track of who owns data we can lump all this together under data governance who owns the data who has access to the data is there any sensitive information stored in this data who's the data about is it about a system is it about a customer is it about a random person it's probably just one bullet point up there but it's very related to just keeping track of the data it's important to introduce these data governance policies again we think that the data catalog solution we're working on will really help with this but honestly like this is an area that we're still not fully mature yet internally and it's something we're working on if you're working on a data lake and you have experience with this we'd love to talk to you about it but it's certainly an important area to consider and I think we're with things like GDPR and CCPA with the California Consumer Privacy Act that's similar to GDPR that's coming out it's becoming very important that you know what data you have everybody's concerned about privacy you have to know what data is in your system and you have to know who owns the data and you have to know if it's subject to regulatory approvals or if it's processes if there are processes defined for getting access to the who can have access to the how do you delete the data how do you update the data when your system starts to scale you have to have processes in place for knowing all this stuff and keeping track of it you're not going to be successful so finally I think the last kind of challenge area that I wanted to talk about is organizing data second and last is organizing data efficiently so this is going to get kind of into the nitty-gritty about Elasticsearch FFS3 for a second again when we initially deployed the data hub when we were scaling up we just kind of wrote data wherever over time we've learned that that was a bad idea specifically for FFS3 we had like so FFS3 if you're not familiar with it you organize data into what are called buckets initially we had like five or six maybe buckets that we just wrote everything to really really big recently we started having issues with our FFS3 server and access started getting really slow the system became very unreliable request for like timey out processes were failing we looked into it and the first thing we saw was that the one problem child bucket had something like 15 million objects in it we started talking to step engineering and to red hat support and we learned that there's kind of this you know if you're in step engineering or support kind of a widely known rule of thumb that a step bucket should not have more than one and a half million objects in it so we were you know order of magnitude over that recommendation that has been fun to deal with I think this was back in like March we learned about this I have a process I can pull it up right now and I can put it around because we're having to move data into different buckets and batch data and combine objects and it has been a pain so my recommendation here is like if you're storing data in S3 we have learned one that buckets are cheap like don't be afraid to create multiple buckets and like come up with a plan for putting your data in these different buckets that's recommendation one it's really easy create different buckets in addition to that or if you can't do that come up with a plan for just like how in like the actual objects what you're going to store in those objects right so one recommendation that we have is for data that is read frequently maybe this is new data as it comes in like in the log example these are today's logs or this week's logs if you need to access those a lot data you access frequently or that is written frequently or whatever store that in maybe dedicated buckets or maybe store that as a bunch of smaller objects and you can access them really quickly right but then in the background or over time or at a regular interval or whatever you want to do go through and reprocess those so that's what we're doing right now is we're taking all these really old millions of objects and combining them into a significantly smaller number of substantially larger objects that way you reduce the total number of objects that you have doing operations like listing objects becomes very fast and you don't really have to access that data that often so if there's a little bit more of an overhead to getting to the data like I think that's that's okay like that's an okay trade off in my book so that's the biggest concern we've had around organizing data in S3 we have had a similar problem with elastic search we haven't fully solved this yet but I think we know how to solve it we've kind of put a bandaid on the situation background on elastic search real quick elastic search stores data in indices indices are split into shards nodes in elastic search cluster has to keep track of all these shards in like metadata right so you can know where data is stored all of that metadata is stored in memory in elastic search and there are limits to how many shards elastic search recommends that you store on an individual node before it starts performing badly that's based on the amount of memory that elastic search nodes has and then the JVM has a maximum amount of RAM heap space that it can keep track of so there's a maximum to the number of shards that you can have on elastic search nodes and it ends up being somewhere in the neighborhood of 600 data stored in our system is stored based on indices and those indices a really easy pattern so you want to limit the number of indices that you have you want to limit the size of indices otherwise it takes a really long time to clear your data a really easy pattern to do in the long example is just create a new index for every single day so a lot of people do that it's really easy the problem with that is maybe you have one data set that generates gigs and gigs and gigs of data in a day and then maybe you have another data set that generates a few kilobytes of data in a day both of those indices are going to get rolled over every day and so over time what you have is a bunch of shards of wildly disparate sizes and that's like it's inefficient right it's not optimal remember that there's a limit to the number of shards that you can have you want to optimize the sides of those shards and you want to optimize the number and the allocation of those shards if you're wasting those precious shard limits with a bunch of really really small shards you're going to have to either massively scale up your cluster or the cluster performance is going to suffer so similar to how you have to plan for bucket placement or object placement in S3 you need to plan for your sharding pattern in elastic search I mentioned a lot of people just roll over the indices every day that's really easy to do it's kind of the low cost easy lazy way better way though that we've discovered is using what's called the elastic search roll over API again this is kind of super elastic search technical if you're using elastic search though you should explore this basically what you do is you specify a size that you want your shards to be and it does not create a new index unless your shards hit that threshold so it allows you to across the board in a really easy scalable way manage the size of your shards optimize the size of your elastic search cluster that you actually have in the so organize your data efficiently and the final area that I want to talk about is just running in containers so containers are obviously by nature ephemeral and sometimes it feels a little bit like fitting a square peg in a round hole the opposite of that though because I think that would actually work a circular peg in a square hole so like the fact that we run a data lake this is all very persistent applications that have persistent storage everything about it is persistent it's a little bit at odds within ephemeral container a couple things I guess three things we've learned that can help you kind of reconcile this the first is use operators so Kubernetes operators are containers that manage an application basically in a nutshell right and typically these operators are developed by people who know the application I have three minutes left I gotta go fast leverage their work don't put all the time into figuring out what it takes to run Kafka in OpenShift when the Strimsy team has already done all that work for you leverage their work do what they've taken all the guesswork out of it before you use what they've done all this stuff is open source then use operators don't reinvent the wheel two is plan for redundancy Kafka, Strimsy if you're working with those on a more generic level OpenShift have a few different mechanisms you can leverage for like fault tolerance or high availability or redundancy that kind of thing three specific ones are with Kafka partition replicas that's not OpenShift specific but replicate your data you can do the same thing with Elasticsearch plan for failure plan for things to be ephemeral create copies of it two is pod anti-affinity Elasticsearch makes it really easy to control on what underlying OpenShift I'm sorry OpenShift makes it really easy to control on what underlying OpenShift nodes your pod runs leverage that don't put all your applications in the same basket put them all on different nodes finally in Kafka you can use something what's called node placement that allows you to put your topics on different nodes I'm close right let me wrap this up you can do similar things with Elasticsearch to control which nodes your Elasticsearch charge run on and then finally the last thing I wanted to talk about were leverage available storage options so OpenShift gives you the ability to use host path mount so if you have something that requires really really fast storage you can use local NVMe storage have that right locally in your OpenShift pods or you can use something like Ceph which is container native storage that works really well so just tailor your needs to the kind of storage you have available to you again we can talk more about that if I need and I don't actually know what time this ends but that's the end of what I have I can take questions if you have them do I have time for questions? cool I have time for questions for the Kafka when you do the normalize where did you get the normalize inside of Kafka like when you write in the topic so yeah so right now we're going to use something called LogStash over time we'll probably use Kafka connectors more and maybe in some cases do like custom Python daemons or whatever language but the pattern is read data so data goes into Kafka in like an ingestion topic our normalizers and then on the back end there's like an official like final level topic that is the last stop before everything gets dumped into elastic search or stuff in the middle these normalizers can run pull data from the front line ingestion topic do what it needs write it to that final line topic where it gets dumped blindly into elastic search or stuff thank you thanks for your talk did you ever consider having like the people that use the hop to manage their own S3 like give me your S3 credentials and it's your problem and not yours so we do give S3 credentials out to individual users so I think you're asking about we connect to their own S3 clusters right we can certainly do that and the tools that we give them like Jupyter Hub or Spark are typically like S3 agnostic like if they have their data stored in AWS S3 they could just connect it to it the problem with making the actual like ours S3 cluster kind of Wild West and letting people do whatever they want is one bad actor and we've found bring down the whole thing so the issue we had is I mentioned we had that one big bucket with 15 million objects but what happened is people were not going to queue to query on that bucket their client would eventually time out but in the back end in SEF that query would keep running the client they'd think oh my command failed I better run this again so in the back end you have all these really expensive queries running that don't finish they bring down the whole cluster I think that's something that hopefully will get fixed this experience has really let us work with the SEF engineering team a lot to try and make this stuff better I think just out of this experience of learning about SEF there's at least two bugs that have been fixed in SEF so hopefully this will get better but right now everybody kind of has to play nice cool alright well thank you everyone


Oh my god. I can't believe I Know I wouldn't want to be treated that way I suggest you guys don't do either but at the same time there's a ruthless efficiency on Just getting the job done and I had a close friend of mine who was dating a girl great gal Just wasn't going where he was wanted to go some of the values weren't aligned Otherwise they would be able to maintain it and she was more than willing to let it ride Not necessarily healthy would not necessarily productive, but that's what people do and at one point He just realized to maintain his own integrity to respect her to respect what they have to respect the memories They created he couldn't do it anymore and wasn't gonna do anymore And what he ended up doing was going up to her house ringing the doorbell. She answers and Says I have you have a key and he was I know gives it to her and he says I love you I care for you and this hurts me into immense degree But we're not right together and I can't do this to you and I can't do it to myself Gives her the key And he goes I'm gonna miss you and I can't see you again. It's gonna hurt too much. It's gonna influence me Hugger gave her a kiss they cried and he left That to me is brutally honest. It's equivalent of shooting somebody that hadn't walked on away. Just not as brutal it has integrity has merit All right Doesn't mean you have to always do it that one I'm also kind of fond of a relationship where you do want to maintain a Friendship after you break up with somebody. I think an exit interview is an outstanding idea. We do it in business Sit down without passion give it a little time and have the exit interview And it sounds kind of silly but you sit down and talk about what with what were the problems? Did you see what were the problems? I say you're not gonna be throwing eggs at each other It's gonna be emotionally volatile. The idea is not to have drama But to come to a relative understanding to have your peace give her the opportunity of peace of mind and respect and Call it a day You know, it's an alternative. There's a myriad of other ways My suggestion is if you're gonna get in relationships know how you're gonna get out of it's kind of like in business as well Your first day of business you better ask how are you gonna get out of business? Which what's your exit plan now that you're planning for failure?  God knows I face this of a woman who realized this isn't working for me It's panicking feeling pressured in any number of ways and how do you get out of a relationship? You don't want to be in and don't have the skill set to get out of it appropriately You get out of it inappropriately All right, and I can tell you to this day she regrets it to this day. She's earned the reputation for what she did All right, I don't like it. I don't like what she did. She didn't have the skill sets involved I can understand that I can forgive her for that But at the same time I don't respect her for how she handled it. She knew better She could have cut she could have acted better and she chose to do the easy path She chose the easy expeditious thing and that was to behave abominably All right, don't do that to yourself. Don't do that to the relationship Don't do that to the good memories that you had because that's kind of all you have All right All right, we've talked about making the cut one of the other aspects on making the cut is this aspect of the notion of Learning through failures need to be able to understand that failure is often where you're gonna find the most growth All right, and you need to develop a personal culture of personal failure without becoming a failure yourself There's a great Ted talk by McChrystal who gave it a talk just about just about that US military special forces commander talked about this idea of Developing a culture of failure without being a failure.   All right The important aspect to hold on to any relationship, especially when it ends is not to hold on to anything past its lifetime Don't hold on to anything that's already gone. All right Part of The analysis here is to recognize when it's time to let go and that is just basically facing reality and the harsh reality on this one Is she's gone? All right, and it's gonna hurt All right, which brings us to the idea that that resistance to reality is a hallmark of suffering All right, there's a difference between being hurt being in pain and suffering suffering something you choose to do for yourself Something that you take on yourself.  All right pain. You can get through hurt You can get through disappointment frustration. You can get through a Sense of loss you can get through these aren't suffering These are real emotions that are legitimate that are tangible to an event during a particular piece of time All right suffering on the other hand is a willful choice to endure something through holding on to something that's gone It doesn't have a point other than the suffering itself. All right, I Also say this I don't think pain is circumscribable. I don't think you can get around Feeling the pain and actually grow and have a healthy life You have to face this. All right part of that is realizing you're vulnerable You're emotional. You have a sense of loss. You've lost something. You are projecting you have expectations for your life Plans events tremendous amount of assets allocated to future projection. That's normal You need to allow yourself that opportunity to grieve that loss to feel that loss to express that loss To be able to let it go if you don't time will not heal all wounds Time coupled with healing and appropriate healing will heal heal wounds. All right Many of them are going to be very very profound nothing more profound than a loss of a family never nothing more loss Than a loss of a wife and a child Okay, and what's worse than even death? These people are still alive. These are choices are making because of a failed relationship All right Again the aspect is that you're not a failure just because something did fail. Don't get stuck in that rut Another kind of aspect I'll hold on to is that even in your darkest hour It only lasts 60 minutes your darkest moment only lasts that long So as you're facing these immensely difficult and challenging situations emotionally Embrace it and know it doesn't last Don't sit down and think this reality is going to be your your existence. It's not it's only going to be your existence If you allow it My aspect is don't allow it to take over your life Letting go of the relationship letting go of the woman is about you not your ex this is something you do for yourself and it's an aspect of Forgiveness you can't let go until you forgive yourself and I'm not necessarily saying you don't forgive her that comes out of Much later state standpoint, but initially you need to give yourself for forgiveness and that is often the difficult tasks We're not trained to do it. We're not cultured to do it But it's something I think is a critical aspect to naturally healing natural growth progression Especially in an adverse situation where you've been done and it's not pretty it never is All right chances are you won't move on Until you're actually ready to all right. That's terribly telling. I've known I I tend to cling on to old relationships Even if I've been years later you kind of hold out hope you kind of hold out for something and often what I Realize is I'm holding out not necessarily for that relationship or that individual. I'm holding out hope for the individual I wanted her to be I'm holding out hope for that relationship in a way in which I wanted or Anticipated to be and the reality is that relationship never existed All right, so often I think we cling to things because we wish they were the way they weren't Recognize the difference between the two One's a real loss. The other is is a loss of anticipation or desire still hurts But the reality is when you're faced with it and you have her back.  All right. It's it's a hormonal addiction. It's an emotional addiction It's a behavioral addiction and she's damn cute You like her in the first place and it's a it's a solution that wasn't of your design This isn't what you want it. You're very very susceptible treat yourself like a recovering addict It's going to be very similar part of that process is to to acknowledge it to acknowledge that you had expectations that were lost Acknowledge that you had desires and wishes that weren't fulfilled All right The other aspect and I think this is part of something that you don't hear a whole lot of is start to accept accountability and accept responsibility for its failure and I'm not saying you're to blame but take on Ownership of the aspects you were What role did you play in this? What role enable this sort of to take place? What role did you play to put yourself to make yourself so vulnerable? Part of that is then Looking at it and saying what can I gain? How do I prevent this? How do I improve? How do I grow? And I think that's what the most critical aspect of the entire growth sequence of healing It's to sit down and say how do I become a better man for having gone through this experience no matter how heinous it was What you ultimately don't want to do is be the same guy you were when you first had that relationship You ultimately want to be a better man What can you salvage out of this burning house that you're going to be able to take forward and make a better life for yourself? Often they're going to be very minor things, but they're going to be significant Sense of pride for having put yourself out there sense of worth for having been vulnerable Willing to engage this to have lived up to your ideals Even if you're egregiously wrong Did you live up to your own expectations? These are things you can't be taken from you.  Either you dump her or more importantly she dumps you and this idea of how do I get her back? The community response is cut or loose new pussy. It solves all your problems. I'm not a big believer in that All right. I'm also not in a big believer is that she's the one I don't think there is just the one there are billions of people probably a billion people on this planet You start doing the numbers Okay, you probably have a lot of relationship potential here All right but the issue is that you found somebody you value one reason or another and you're not quite ready to let it go or it's Re-emerging from a distance path the question is is is when is it a healthy to re-engage in a relationship and resurrect an old relationship?