So what we're going to do is take a few of the main objections or common objections to capitalism. And I am using the term capitalism here, which some people have objected to. They say, well, I prefer that you use the term free market. And I understand that because sometimes when people think of capitalism, they think of crony capitalism, which means that you've got a business that is using the government to somehow squelch its competition or gain subsidies from taxpayers or the like. And so we're not talking about crony capitalism here. We're talking about capitalism in the way that Ludwig von Mises used the term, which is to mean free markets. And I do understand that the idea that the term free markets is one that has perhaps less ambiguity. So that's good. Well, let's, without further ado, go into the first of the objections. Capitalism exploits the poor. And you've seen this before. This is standard fair in history classes from middle school on up about how the Industrial Revolution impoverished urban dwelling families and led to children working in coal mines and textile mills and grievous injuries in the meat packing plants and so forth. This is the kind of thing that you can expect from capitalism, that it makes the poor worse off. I think that the problem here is a little bit of a memory problem, that we forget what life was like before the Industrial Revolution. We forget what life was like, in other words, before the market economy began to be predominant in Europe and then eventually elsewhere in the world. We do see that the poor are much better off in countries that have a higher degree of market freedom. Matt Ridley in his book The Rational Optimist says, today of Americans officially designated as poor, 99% have electricity running water, flush toilets in the refrigerator, 95% have a television, 88% a telephone, 71% a car and 70% air conditioning. Cornelius Vanderbilt, one of the wealthiest men in the world a century ago, or a little over a century ago, had none of these. If you want to make the poor better off, then free markets are the way to go. Now, what about the share of income that goes to the poor? We see a lot of attention given to inequality. Frankly, I think that's because that's one of the only hooks that the anti-market crowd has left. If you look at the percentage of wealth that the poorest 10% have worldwide, it is not that much different from the least free quartile of nations and the most free quartile of nations. This is from that same Economic Freedom of the World report that I mentioned earlier. You can see that no matter what level of economic freedom the poor have about 2% to 2.5% of the income earned in the country. However, 2.5% of a small number is a lot smaller than 2.5% of a large number. If you look at the income level of the poorest 10%, you get something very different. On the right, you see the most free quartile of nations that are indexed in that Economic Freedom of the World report. Not all countries made it into that report. For example, Cuba is not there and North Korea is not there. I could guess which quartile they'd show up in. You can see that if you are in the poorest 10% in the most free nations in the world, your income is likely to be about $11,400 a year. Whereas if you are in the least free quartile of nations, it's more likely to be about $1,200 a year. Big difference. Here's one that I have heard frequently, including from some close family members of mine. Socialism works in Europe. You heard this during the presidential campaign this year. You've seen a lot of talk about how great it is in Denmark and how great it is in the Netherlands. They're socialists and therefore we should be socialists too. Well, again, I'm going to go back to that Economic Freedom of the World report. Here are the top 30 nations out of I think about 160 or so in the world that are indexed in that report. You can see that there's the United States, number 16. You see a lot of European nations that are in that top group as well. The Netherlands is there at number 30, Germany's 29, Norway and Luxembourg 28 and 27 and Denmark's number 22 and so forth. So that puts these European nations well within the top quartile of Economic Freedom. And what we really see is the difference between the European economies and the U.S. economy is the kind of socialism that is there and not so much the level of socialism. The United States on the 0-10 score of Economic Freedom has a score of 7.73. The Netherlands is 7.48. Denmark is 7.58. And within that aggregate number, I really shouldn't use the term aggregate number in an Austrian economics conference but I guess I did. But the Netherlands is number nine well above the United States on their legal system and property rights. They're number 15 on their monetary system. They're number eight on the freedom to trade internationally. Now they're very low on the overall size of government. They're actually number 155 out of 157. So the Netherlands has a very large government, a large welfare state but that is offset by some of the salutary effects of other areas of relative freedom. So the United States, by comparison, is number 29 on the legal system and property rights, number 38 on the monetary system and number 74 on the freedom to trade internationally. So it's a matter of style more than substance as far as how the United States differs from these other countries. So for a presidential candidate to say that socialism works somewhere else, well that's not quite accurate. And if you look at the lowest group within the same index, you see a lot of countries which would not be very high on my list of places to immigrate if I ever left the United States. Number 157 in the last ranking goes to Venezuela. And if you haven't heard about what's going on in Venezuela, apparently the socialists there have managed to create food shortages, which takes some doing but they've managed to do it. Also Syria, Chad, Libya and the Republic of the Congo make it to the bottom of that list. You don't see a lot of European countries there. I think there's one on the list if I remember correctly. Ukraine, yes, tied with Ghana. Ukraine has made very little economic progress since independence from the Soviet Union. Georgie Itte, who's the chairman or president rather of the Free Africa Foundation, says Africa is poor and you see a lot of African nations represented here. Africa is poor because she is not free. That's sort of his motto for explaining the African situation. Africa is improving a great deal, much more so than many people recognize, but the reason for their poverty is not some sort of lack of foreign aid. It is a lack of free markets. So number three is capitalism is unjust. First of all we have to decide whether we're talking about inequality or the well-being of the poorest fraction of the society. As we saw in one of the previous graphs, the poorest 10% get about 2 to 2.5% of the income, regardless of the level of economic freedom, but the 2.5% of a large income makes the poor relatively well off in countries that have more economic freedom. This is, as Hayek said, due to the fact that when a nation gains economic freedom, the new production that takes place tends to be directed at satisfying the desires of the masses. If you were to go back to the period of feudalism in Europe, we found the top .01% living with a great deal of wealth, a vast majority of people living in what we would clearly see today as poverty, but as the market system began to take root in Europe, the real beneficiaries of this were the poor in the middle class. Once the rise in the position of the lower classes gathers speed, catering to the rich ceases to be the main source of great gain and gives place to efforts directed towards the needs of the masses. Those forces which at first make inequality self-accentuating thus later tend to diminish it. Now, we do see some increased inequality. For example, there is increased specialization, global trade, and so forth that leads to a kind of superstar effect. So, if you look at, for example, a microcosm of that would be the payment of actors. There are a few actors at the very top of their profession that make very high incomes, and then there are a lot who are waiting tables while they're waiting for their big break. There are very few that are playing Major League Baseball and earning huge salaries, and many, many who are laboring in the minor leagues waiting for their big break. And what we see is a kind of bifurcation of incomes as some people manage to sell their services to a very, very large audience. There's increased immigration. Immigrants tend to have lower incomes, and if you have a country with increased immigration, you're going to tend to see more inequality from that factor alone. Also, we see some, I hesitate to use this word, but sociological effects as well. High earners tend to marry each other more than they once did. One set of my grandparents was comprised of a college graduate and an eighth grade graduate. You don't see that nearly as much anymore. College graduates tend to marry college graduates and so forth. However, we do see some evidence of decreased inequality in some cases. IQ score spreads are shrinking as the low scores are rising. Matt Ridley points out that British aristocrats were six inches taller than the average height in 1800, and today that class is less than two inches taller. And then there's more data. I mean, this is not the only pretty thin evidence, but there are a lot of other data that I don't have time to go through here that indicates that in some cases inequality is decreasing in some important dimensions. Tanner and Hughes pointed out in 2013 one of the problems with welfare systems, which is that it creates more inequality in some cases. It discourages people from earning income. In 35 states, welfare pays more than a minimum wage job, discouraging people from taking minimum wage jobs. We could talk later about the minimum wage. I think we've got one of our speakers devoting an entire 45 minutes to that subject, so I won't dwell on it. But in 13 states, welfare pays more than $15 an hour. And so for many welfare recipients, especially the long-term dependence on welfare, there's a disincentive to work. One of the most interesting graphs that I've come across in the last couple of years is this one here. What I want you to pay close attention to is this thick, dark line here, this stair step here on the bottom graph. What these researchers did is look at the percentage of your income that you got to keep net of lost benefits. So if you look at the tax rate, that's the pale blue line here, and of course the tax rate is progressive, and it goes up as you earn more income. On the horizontal axis here, you've got the earnings, and then you've got the implicit tax rate here. The pale blue is the actual income tax, and then as you earn more income, if you're starting down here at the low end, as you earn more income, you're going to lose some of those welfare benefits, which from the perspective of incentives creates a disincentive to earn more income. So if you earn an extra dollar, you get an earned income tax credit, which acts like a negative income tax, and so that's offset and more so by lost benefits. As you earn more income, you lose some of your eligibility for these programs. So that creates a hump here where for a certain income range between about $17,000 a year and about $20,000 a year, the implicit tax rate, your income tax plus the benefits you lose mean that you get about three cents better off for every dollar you earn. So if you're someone who's sitting down here at this level of income, basically you're looking at a tremendous increase in earned income that you'd have to make before you would become appreciably better off. And so the upper diagram here shows, with this dark line, shows how much your disposable income would be as you earn more dollars of income, and this is for the United States. So between about $10,000 and about $22,000, this line is almost flat, which indicates that if you double your earned income, you go from working 15 hours a week to 30 hours a week or something like that, you double your earned income and you're not really appreciably better off. What that does is accentuate inequality by discouraging people from earning more income. So you have a group down here that's discouraged from hurtling this mountain here and others who are on the other side of it who have a stronger incentive to increase their income. So governments themselves are increasing inequality by the incentives they create. I had another question to ask you, and the Plickers thing was not cooperating. I'm still in amateur with it, but you see some of this, and this was a complaint a few years ago and still it seems to recur when the Occupy Wall Street group was complaining about the payment to CEOs and so forth. And you see graphs like this, which indicate that the ratio of pay between CEOs and workers is very high, and it looks like at the time this was produced, the ratio was about 300 to one. So CEOs were earning about 300 times the average worker pay in the 350 publicly owned firms with the largest revenue every year. And so this sort of thing makes it into the media quite often. However, there are a lot of misconceptions on CEO pay. First of all, this is the 350 publicly owned firms with the largest revenue every year. So of course that's going to tend to be higher. And you do see this trend is up. Matt Palumbo last year in a Mises Daily article pointed out that firms are getting larger. The average size of a company on the S&P 500 is getting larger. And according to another study by the NBER, the increase in CEO pay between 1980 and 2003 is about six times, and that's consistent with about a six-time increase in market capitalization of large U.S. companies. So it should not be surprising. What is surprising for a lot of people because they focus on the large firms and the CEO pay of those large firms, these are kind of superstar CEOs to go back to that superstar effect, people forget that that's not the typical CEO pay. Now I'll ask you what my software wasn't allowing me to do in a high-tech way. I'll just ask you in a low-tech way. What do you think the average CEO in the United States makes every year? 300,000? 250? 100. Okay. This is not an auction. A reverse auction. All right. Well, 178,000, so you're very close. That's very good. A little more than dentists. Okay, so most CEOs are not working for those 350 largest firms and the pay is not exceptional if you look at other highly skilled individuals, doctors, dentists, orthodontists, and others that we know they have a comfortable living, but we don't think of them as being the super rich. And so that's important to recognize. Moving along, what about this argument? The capitalism doesn't promote real happiness. Now, this is largely attributable to a study that appeared about 1972 or 73 that seemed to indicate that the higher-income countries did not have happier people in them. And unfortunately, that led to a myth that has persisted to this day that high-income countries are full of people that are on various kinds of antidepressants because they are generally unhappier than people who are much poorer in a material sense. Two much more recent studies, I think they were both done about 2008, and I can provide you with the links or details on that if you like afterward, showed that, in fact, that's not true. The earlier studies were flawed with very small sample sizes, and I know using the term sample size, I'm two strikes now, so. But they had very small sample sizes, and the actual fact is that we see a correlation, a positive correlation between GDP per capita and what these researchers are calling mean life satisfaction. The size of the bubble here indicates the population of the nation, and you can see that there's a pretty clear positive correlation there between higher GDP and higher mean life satisfaction. This should not be terribly surprising. If you have higher incomes, that means that you can more often resolve painful illnesses. You can alleviate distress of many kinds. Infant mortality goes down. When people's children don't die in early childhood, they tend to be happy about that. And one of the positive effects of higher GDP is that we get better mother-baby care and we get other medical interventions that allow people to survive who otherwise would not and allow people to survive with less pain than they otherwise would. They get to communicate over long distances with their loved ones, travel to see their loved ones with greater frequency and ease. Many, many things that allow people to have greater life satisfaction as their incomes go up. So this kind of argument that people are less happy in more free market countries does not seem to hold water. I hear this one quite a bit. This number five here is that capitalism is aesthetically obnoxious and environmentally irresponsible. Mises in his book The Anti-Capitalistic Mentality said only romantic pre-possession can induce an observer to ignore the fact that more and more citizens of the capitalistic countries live in an environment which cannot simply be dismissed as ugly. And I had an article I published some years ago, had published some years ago on the coastal areas of South Carolina and North Carolina. And I just came from a family trip to the coast of South Carolina and of course if you draw near the coast you find that there are a lot of retailers and fast food restaurants and condominiums and other things. And there are many including myself who enjoy seeing natural landscapes, palm trees and beaches and fewer people. I mean I would like to go out on a beach that has not 50 people per square meter but has more space. I enjoy those empty spaces. I went out kayaking recently in some of the salt marshes of South Carolina. I enjoy the silence. Got within a paddle's reach of a great blue heron. All of that is something I enjoy. And people somehow have this idea that capitalism is going to lead to less of that. I can point you to the article I wrote on this which indicates that no, that's not in fact what you would find in a more free market society. And in fact you see a lot of this kind of thing here in societies that have fewer free market characteristics. Now beauty is subjective. So I'm not arguing for some kind of objective standard of beauty. I will simply share my perspective on this which is I think these buildings are ugly. You are free to think otherwise. You may look at this and say isn't it wonderful that we have low cost housing? And yet I would contend that even low cost housing in free market, more free market economies tends to be less unattractive. Well a few years ago you may remember we had an oil spill in the Gulf of Mexico and every so often there will be some environmental catastrophe of that sort that leads people to complain that capitalistic economies don't protect their environments. I think this is one of the problems that millennials face in trying to understand what the connection is between the environment and markets because they don't remember as readily the environment of Eastern Europe under communism. The little Lada cars and I can't remember the other Ladas were very poor quality, smoky, polluting vehicles but they were approved by the state, built by the state and Germany and some other Eastern European countries were environmentally crisis zones after many years of the state running things. I don't think it's fair to say that the state is some kind of friend of the environment while the capitalist is the foe. That's simply not the case. Now looking at the BP oil spill in particular, Matthew Novak pointed out that the government passed laws that gave the oil companies incentives to drill further offshore where the risk is higher where if you have an accident it's much more difficult to deal with it in thousands of feet of water than to deal with that in shallow water. If you look at the percentage of oil production in the Gulf of Mexico from depths greater than 200 meters you see that between 1995 and 2003 that share for crude oil went from 26.4% to nearly 70% because the tax rate on deep water oil extraction was far lower than the tax rate on inshore shallow water production. So the state sort of set the table for this kind of catastrophe by encouraging deeper water, higher risk kinds of oil extraction and then too in some cases where there's high risk of environmental catastrophe the state implicitly or explicitly ensures the private firm against that kind of catastrophe. So for example nuclear power plants, if they have a catastrophe they're limited in how much financial liability they have. The government is standing behind these firms making sure they do not have to bear the full cost of the potential catastrophe. What about this one? Corporations are full of scandal and extortion and I'm not going to say a lot about this I'll simply mention war, government-induced famine, genocide, corporate welfare, political scandal, bribery, corruption, police brutality, deadly regulation, fascism and more war and so on. I think this might be, is that Dresden or is that, well the fact that I have to ask which city is this indicates perhaps part of the problem. This is not the result of a tsunami or an earthquake. I think this is the result of a firestorm induced by heavy bombing, by the state. And so I was on the Facebook page of my homeowner's association recently and I was looking at some of the comments of people who were complaining that somebody cut down a tree without notifying the neighborhood and they're not cleaning up the river path as much as they would like or they didn't get approval from the community before they decided to do X with some piece of the community property. I started thinking homeowner's associations are such petty organizations in some cases. I mean they just, somebody cuts down a tree and people get up in arms about it. And then I thought, well that may be but my homeowner's association is not going to tax me to go invade some neighboring homeowner's association or I don't have to worry as I'm walking around the neighborhood that I'll be accosted by some representative of the homeowner's association who is eager to extract something from me. The worst thing they can do to me is put a lien on my house when I try to sell it if I have not paid my very modest homeowner's association dues. So there's a vast range of difference between this sort of micro-government that we might call a homeowner's association and the kinds of things we normally think of with the state. People tend to think of private firms as being extortionists and I ran across this a couple of years ago when Uber decided to charge, I think it was an extra dollar per fare as a safety fee to account for certain regulatory constraints that they had to deal with. So this website complained about this, I'm not sure exactly who Pando.com is, but they said it's entirely possible that people will recoil at Uber's deployment of the well-worn libertarian strategy of forwarding the costs of compliance with regulation on to customers as though that's a libertarian strategy not just an inevitability of imposing costs on firms. This is regulation imposes costs on shareholders, the firm's owners, it imposes costs on the customers, costs on the suppliers, and so forth. It's not some sort of secretive libertarian tactic to try to get people to object to regulation. Of course in the last two years since this was published, Uber has done very well, it does not appear the customers are recoiling in any way. I'm running out of time, so I'll be quick with this argument that capitalism is racist and sexist. Now in a free society, people are allowed to have whatever taste and preferences they like. If you prefer those Kazakhstan high-rises, then that's fine. You can have whatever taste and preferences you like. But ignoring productivity in order to favor one group over another may come at a cost. So if you decide you don't want to serve a certain demographic of customers at your restaurant, you're going to pay that cost as an entrepreneur. You're ruling out a group of customers whose money is just as good as the others. And so that will come with its own penalty. In contrast, government intervention can shield people from the real economic consequences of those kinds of discriminatory actions. So if you, for example, put a price control on rent and prevent people from... prevent building owners from raising the rent on their apartments, then you have many people who are willing to rent the apartment for every available spot. And the landlord can then pick and choose which potential tenant the landlord wants. And that is right for a kind of discriminatory practice. Whereas without rent control, there would be far fewer potential tenants for every available apartment. And it's much more likely that a discriminating landlord who's discriminating on some characteristic that has nothing to do with the quality of the tenant would then have a vacant apartment and lose revenue because of this. This is a picture from about 110 years ago of a street car, or a street, rather, in Jacksonville, Florida. There's an article by Jeff Jacobi on Kathy Hayek and another by George Leaf. I can give you the links on those later if you like. Jacobi writes that the owners of municipal transportation systems like these street car companies, their private companies, actively resisted segregation. Now, they may not have had some kind of lofty commitment to equality or integration, but segregation cut into their profits. It hurt their bottom line. And so the manager of Houston's street car company complained to the city council in 1904 that the street car companies were being forced to haul around a lot of empty space because the city government had this rule that whites and blacks had to be separated on street cars. So in many cities, also the segregation laws are promoted boycotts or provoked boycotts of street car companies which further cut into their profit. So in an article in the Journal of Economic History by Jennifer Roback in one southern city after another, she says private transit companies tried to scuttle segregation laws or simply ignored them. In Jacksonville, Florida, a 1901 ordinance requiring black passengers to be segregated went unenforced until 1905 and then the state legislature mandated segregation statewide. This was passed against the opposition of the street car companies. There are many other examples of this kind of thing as well. In fact, the company's hostility, the street car company's hostility, the segregation was so well known that when a group of black citizens challenged the law in court, their attorney said, we're not associated with the railroad lines in Jacksonville. We're doing this on our own because it was so well understood the street car companies, private firms did not want to segregate their own street cars. And there's much more of this. I will leave it to you to investigate these works on another criticism of capitalism which is that capitalism is prone to bubbles and panics. You will hear about the Austrian business cycle theory this week and the argument that the state is a promoter of bubbles and panics rather than a guard against them. I'm also out of time on this idea that capitalism won't pursue important projects. On the way up here, though I will say one thing, I was listening to a book by David McCullough on the Wright Brothers. And one of the interesting things he points out is that the federal government through the Defense Department was supporting a competitor for the Wright Brothers, Langley, and an airplane which you can see there on the, well, I call it an airplane, it's in the process of crashing on the upper left there, which he called the great aerodrome which was produced right about the time when the Wright Brothers were producing their aircraft. This cost the taxpayers of the United States about $50,000. The Wright Brothers plane over here, which you see in flight, cost about $1,000 of their own money and they managed to do the project while operating a successful bicycle business in Dayton, Ohio. They refused government assistance and in fact, when they pointed out to the federal government that they had developed a useful aircraft, the federal government sent them a form letter in response and said, no thanks for not interested. So anyway, I will stop here since I am out of time. Thank you very much for your attention.


The next talk will be given by Claudia Agosti. He came from security, then moved to privacy, and then he saw what is happening in this social media area with all this personalization that is taking place so that everything is very nice for you or something like this. Well, so a lot of people are a little bit wary of what is going on there, but it's not really transparent so far. And what he does is to devise things, tools, algorithms to explain what is going on there with your privacy or personalization. Claudia Agosti. Thank you. Well, beside Claudia Agosti, there is an entire team of volunteers. This project began one year ago. The project began one year ago. We collected volunteers and we reached a beta stage. I was supposed to release the 1.0, but nobody is actually paid to develop this program, so we are doing it in the free time. The title is The Quest for Algorithmic Diversity, we will see later. I'm not a TV English speaker. I will try to speak slowly to be understood. The talk is intended to last for 35 minutes, so we have time for questions and answers. The website, Facebook Tracking Exposed, is also the name of the project. Exposed is a true top-level domain, so feel free to press enter. There is not a .com, .org, just enter. We address two big problems. I can begin by asking how many people are on Facebook, but we realize that it doesn't really matter. What we are going to criticize here is the algorithm of hegemony, so the influence of the social network to frame the perception of the reality. Every person perceives based on their own bubble, their own sources, their own activity, their own preference, and this implicitly gives to the user a personalized experience. The social media has, as main goal, the one that keeps you in the social media, active, engaged, and that became, you get only what you really want. Considering that many persons are shifting their own information model from official newspaper or from, I don't know, maybe more social interaction to the social media, that implicitly is a chunk of perception of the reality that is influenced and decided by an algorithm. The second point is the amount of data collected by Facebook can really help to understand the society. If the society can check each other, can check what is happening. At the phenomena, I don't speak about looking at the individual, but it can be a way to understand trends and how, in generally speaking, the person changes their interest, their belief, their action, their will. But this is not possible because Facebook privatized the information and you can get only access through certain limited API. This project want to offer a way to understand the problem, not yet a solution. There are some solutions around that are experimental, but this try to provide a tool to understand what are we talking about and what is the problem. The concept is a web extension that run in your browser. At the moment, it works only for Chromium, but it's in progress, the port for Firefox. If someone want to help, there is an issue on GitHub. The extension looks at your Facebook activity because it want to actually scrape the post you are receiving. Only the public post, the one shared with the world, so not scraped the things shared with friends or for restricted audience. Every time you access on the news feed, that is a timeline, we have 150,000 entries currently. A timeline as many posts, they are the impression. The impression can be private or public. When they are public is because it's a public post and we get the entire HTML snippet. In this way, you are collecting for yourself or for the network your actual timeline. What you saw, you can call it back later. This permit to start to begin to understand what are you receiving because sometimes the needs of the user became, I want to be more informed, I want to know what Facebook hides, I want to escape my bubble, but those are quite abstract needs, are not practical. Start to begin understanding your information diet, I call it sometimes, or what actually you are receiving. This is an example on 20 days, and this user was receiving a post from a certain amount of sources based on the color. But those sources, the source is a metadata, a metadata that you extract from the HTML. There are also other metadata you can extract and associate. For example, you can start to understand by sentiment, by keyword, by language, and make more complex visualization that can give you an insight of how you are informed because at the end of the day, the algorithm has a big role, but you have a bigger decision in making your information diet diverse or complete. But then we see also the power of the algorithm and how some kind of repression can be exerted or some kind of influence can be exerted. So a poster can show up on the top of the news feed, constantly, despite not being chronological order. This has been done when Facebook in the 2013 or 14, I don't remember, released the experiment on the emotion contagion. So 600,000 persons have been used in an experiment. Half of them were keeping seeing sad, angry posts on top of the news feed, and the other were just seeing happy posts, just to check how the humans react to this kind of stimulation. And that's, in one side, is the example of the algorithm abuse. Is that okay that I happen, that the user are at the lab to do this kind of experiment? And is that okay that my perception of the reality changed because something keep appearing on top and my attention economy get polarized by this content? In the moment you take a copy of what you're seeing, you can apply analysis and start to understand if it's happening or not. If it's something very frequent or something rare. And also, post maybe they just show up once, but then if you do not interact, that is appeared. We know that exist. We know that this is happening because that is how algorithm works. But understanding how much can permit us to also make a more precise criticism to Facebook. Because in this moment Facebook or the partnership for artificial intelligence are starting to saying maybe we should be transparent or maybe we should give accountability. But there is not a terminology, there is not a political needs, there is not a perception of how the algorithm actually can impact in your life. Having an insight that begin from your experience is the first one to elaborate what is okay, what is not. This is an example on a timeline. Here the post are ordered by the number on the left is based on chronological order. So the first one is also the last one published. The second is the fifth in time and the second third are below. The yellow one is highlighted and you see for example, this is another post, but you see for example that it was remaining in the same place in the same position of the second timeline 10 minutes later and then was going down. That can be an expected phenomenon. It is justified because a generic Facebook lawyer but also we know that we need algorithm. They are necessary to sort it out all the information we are accessing. But algorithm define your priority.  And this priority decision is a decision we are not taking. It's something that Facebook is taking for us. Investigate on the algorithm is not so simple. Other experiment has been made. Sometimes this experiment try to show you just who is advertising on your timeline or who is writing to your news feed. But what we are receiving is composed by five high level variable. How the supporter use Facebook because if I keeping accessing to the social media I'm populating my data with a lot of access. If I keep accessing like in an automatized way or all the day, my possibility to see information is higher compared to someone who access rarely. Which source you have because you decide your own bubble? What the source share? Because your bubble implicitly may be polarized by some external event or by their own. The Facebook algorithm that is something we are trying to understand and the supporter profile. So all the kind of your backstory based on your conversation in messenger may be your account of Instagram linked, your likes, your shared data. The data you shared is something we have not. We just look what are you receiving. So based on those assumption, it's clear that we cannot moiter what you cannot moiter what you receive or the researcher cannot do analysis simply. Thinking that based on our small amount of users we can understand how Facebook is working. So we start to run some text, some tests. If those are variable, we can try to reduce their variation in order to test the algorithm or exclude the unknown part. So we made four fresh user with zero friend. They were following the same 12 national media. They were accessing Facebook at the same time in the sense that it was once per hour. They were doing automatic scrolling for three minutes. So access once, three minutes of scrolling, wait for 57 minutes, and then refresh. In this way, you are following these 12 sources and you can see if they publish something new, if the things is appearing for the user in the same way. But we also have made some differentiation. Specifically, every person, every user-bought character has some likes of characterization. So over four users, this test has been made before the French election. One was liking Le Pen, the other Macron. Another was mostly liking a German rapper and the other was liking DR. So a very different kind of subject that may be in the profile will arise. What we saw is that despite that they were accessing frequently, they were sharing the same news media. You see here the post ID, so the post that they get published. This is the user ID. And this the hour, the hour in which they were accessing. What can be interesting is that despite we try to uniform their access, some news was not showing up. This user, as accessed for five time, the post was existing at 15, yes. But in that user never show up. You can see the same in this line. And those are posts that despite they were keeping accessing Facebook, despite the post was available, was not showing up. Why? Algorithm decide. That's, I don't know, it's not exactly OK. We can do different analysis. Also, I keep saying we, but we will see the project is quite open, needed for collaboration. So when I say we, means you with us, can do analysis. For example, to see, to analyze the comments. We know that comments provide a certain kind of frame of the news you are seeing. When you have many comments, you can look the previous, look the next, but you have some preview. And those previews are all the same. How much this preview can influence your judgment on a political event? To do that, we have a structure. And that's just if I also, why we are scraping the entire HTML. We are receiving the HTML that can be represented in this way. That is what we are receiving. And then there is a batch of process who extract for every metadata we can be interested, only the specific metadata. This because Facebook changed the HTML structure quite often. So you need to catch up quite fastly if they updated. For example, if the rainbow reaction get add, you can update your parser to extract also the rainbow reaction. Or this project begun when, for a certain moment, Facebook made the flowers. The flowers to say thank you. In that moment, the HTML structure changed a little bit. Here you see the metadata extracted. In blue are the parser that have run over that HTML. And in black are the metadata attributed. This has been made because the HTML structure was changing quite often. At the beginning, during the alpha, we implemented in thank you to take picture to the slide. But I have to tell you, they will be online. They will be online on the website, Facebook.tracking.exposed. But if you want to take a selfie with the slide, that is unique. It will not happen every time. I can thank you. This is a mistake that happened during the alpha. We were collecting a promoted post in red and feed the post. Happened that Facebook changed the structure. And we start to see the promoted post as a part of the fee. And the feed the post start to be ignored. So we actually collected it. We broke the data during the alpha. That has made us learn that the parsing has to happen server side for that reason. The parser has to be contained as much as possible. They are based on, they are implemented currently, not JS with Cheerio. But there are also some experiments made in Python because you can just talk with an API that say, give me the HTML that has not this field. You get the HTML. You extract your field processing your code. And then you update the HTML structure with the new metadata. The metadata can be something extracted from Facebook. So the name of the source, the number of likes, or even something that is not present. The language, the sentiment, the entities, the complexity of the word used, the presence of a speech or a specific keyword. Or eventually, if you have a list of sources and you want to attribute their own trustworthiness, you can compare the source with the source list. And the metadata that you append to the database is the trustworthiness. In this case, it's just based on a CSS selector because that is most of the time what the parser are doing, what the parsing are doing. Algorithms are a social policy. So that is the theory. If algorithm can influence how society works, that means that they have to be open to public scrutiny because it's in the public interest. Until we do not understand how the algorithm, as a leverage, also the political need will miss. Also the kind of request that we can do as a user or a citizen of a federation or a country. We can do to Facebook are limited because we are not understanding. That is a problem that is getting perceived much more in the last year. With Zuckerberg announcing that they want to do a global community, yeah, good, a global community, but with an algorithm. It's like all the citizens are just one mono citizen of Facebook. We have pretty much different needs if the needs are absolved by our priority, by our values. Those values and those priority has to be embedded in the algorithm. We cannot expect that Facebook with one single algorithm can satisfy everybody. The effect has also impact in who use Facebook to communication because you feel that you have to be present in Facebook because otherwise you cannot communicate. But Facebook implicitly force the person to communicate in the way they want. This is a journalist Kurt Gessler of the Chicago Tribune and here lies that some posts were not surfacing anymore because Facebook decide that the video has to be more promoted. So they were just writing text and that has caused downfall of their own income and to the access. Because what the media is quite often complaining about is that the user are not accessing to the news media through their own homepage but only through social network. News, recent news is that Facebook want to change the news feed in favor to website that load faster. So at the end of the day, the algorithm is a way to incentivize person, users or community to communicate in the way Facebook want. And that is the algorithm hegemony. And that's, I don't know, I'm not seeing up to be part of Facebook land but only through Facebook social network. Imagine that in the future, the social media will just be storage of data and you can decide your own algorithm client side. The user can customize their own algorithm, exchange it with other, remix it, resell it if they have special values or change it when you need it. Because if I live in Italy, I wake up all the days looking on my news feed because I want to be normally informed, I can have an algorithm. But in the moment I become a tourist or I'm migrating or I'm looking for something new or I broke my legs, my needs are changing. If the needs changing, also the algorithm has to reflect my new needs and I want to be in control of it. That's who's the part about the algorithm hegemony. Now thinking that Facebook is a sort of mirror of the reality. Clearly if we can, as a society, access to this data set, we can have insight and understand something more about us, understand or judge what is okay, what is not and take more informed decision. But in the graph, in the big network, you cannot have a single way to observe the network. Every user is an observation point. So if we start to use the extension, you start to observe Facebook, what we will get is a vision of the world that is only a subject of the hacker scene. Because most of you belong to the hacker scene. The observer is never neutral and all these observer are also always intermediated by the algorithm. So at the end of the day, we are not really reading the society. We are just reading a subset of it, partial and biased. But still can be enough to do some analysis or to understand a little bit more what Facebook can do and eventually ask collectively to have more access to this data because it can be in our interest. That is the goal somehow. This project is not intended only to empower user but also to rise criticism and show Facebook that there are other ways that they are not exploring that maybe can be done. So the data collected, the data we are collecting, they can also be used by a user that are not yourself. In this moment, we have some guidelines that I protect the individual data as much as possible, listing the user needs in the sense that if we want to provide visualization or analysis, this has to be done in the interest of the community who feel a problem. Not because I believe that the fake news are the one from Brevard. I will mark that the news feed with Brevard are red. This is a sort of subjective judgment. It's not something we want to do but we want to enable person to implement their own value in the analysis. Data transparency policy. So try to explain as best as possible when a data arrive in our system, how it's treated. We know that this data is processed by parser, can be mixed, can be reduced, can be a nice experiment, try to do the best data transparency policy because this can be an example for other company who actually have to do a data transparency policy. Data transparency policy means explaining all the place and the reason where data is stored, if we can do that, can be a way to show how can be done in a more human rights perspective way and give back the agenda in the user end. So try to make the user experiment their own algorithm and make feel how it is if you can customize it. This is the pipeline that is going to be completed. Impression and HTML, so the impression are just the entry of what you, how many posts you saw and HTML are only for the public. The parser extend the metadata and the metadata can be accessed in three different way. If you are the owner of the data, you can access to your own data. Eventually you can share it. This means that a nice functionality can be that you can share your own personal point of vision, point of view with a person that you trust. A parent, a partner, a colleague or a professor can be a way to understand that everybody's perceiving a different reality and check each other. Anonymized aggregated API. Anonymized data from a graph is hard but some kind of API limited and analyzed can be done that will avoid at best correlation. That can be a way to analyze phenomena. And then researcher who accept an ethical agreement, for example, you will never use this data to create a profile of the user. You will not use this data outside of the research scope. Some logic on publishing who is accessing this data as a transparency are the way we think that the metadata chain can be queried. As a doc.go care to make us remind, anonymized data is very hard. That's why instead of releasing anonymous data, we rather prefer to make the MapReduce code or the API implemented in the server so it can be privacy analyzed and be sure that no correlation element get leaked. It's already happened that a researcher, Sylvia Puglisi, used the database as a verification. She published this paper to test her algorithm in a de-anonymized graph and then has been tested on this dataset. So this mean also other researchers that are not just belonging to a technical field can access to this metadata chain. But that's as to complain with the ethical agreement. We will find the procedure, but that's what we are open for. Next step, what time is it? No problem, there is time. Next step, finding users, writers, analysts and developers, I'm afraid in this order because developers are very important. Mostly we are developers, but what we are needing now are mostly person analyzing the data and extracting insight or start to understand how this huge amount of metadata can be, you can make a sense out of it because it can be very a lot, can be more than what you can expect. Users, because they contribute to different point of view and writers because to engage users and to create a vision on why this is a social problem and as to be addressed, they are necessary. Developers, welcome. In the sense that most of you probably can help us for the technical side. We have a GitHub repository and we are very open to implement new technology to do the analysis or to manage the metadata. We have to complete actually the 1.0, so it's the right time to join the project. Explore the diversity of the needs in the sense that we know that the Facebook graph can contain many insight, but we have not the habit to use it. Normally, the Facebook graph is used to do marketing research or to target the user to communicate to them when they are more vulnerable or more interested to buy, but there are some socially useful way, maybe, we have to find them. Whenever some visualization, some analysis, some insight can be implemented by a community or by some experiment when this can become stabilized can be implemented in the main software so all the user can see it. Some practical idea to explain what can be done next if you are interested. Help researcher in figure out a way to use it. So if you know some researcher that is doing analysis in media and they need to have access to Facebook, accessing to Facebook through scraping is hard. Accessing from Facebook from their own profile is biased because you are accessing through your bubble or through your page that you know. Accessing Facebook through the API is limited by the API itself. This can be a way to do a new set of social science research. Run some experiment. For example, we made experiment before the French election with four users created from scratch but you can run the experiment among your group of friends. Maybe with your friends you are following some of the same source. You start to monitor your own and your friends the same and then you compare how much they are seen by you and then you can run some automatic user to keep accessing to this source to be sure to get the 100% of their publication. Write a new parser. For example, we have not yet a parser to extract the number of likes or the number of comments. It's pretty simple, it's just a number but I was trying to keep the simple task to be done by new commerce so it can be easier and make maybe a community of developers. Use the API, the API to query the data or ask for a new one because in this moment we are just a small team we lack of this kind of testing and imagine how to visualize data. When you have so many metadata it's quite hard to actually make meaningful visualization. If you have experience in that is helpful. Tomorrow at the Italian embassy that is at the corner, yeah, where the grappa is. But at 10 a.m. 50 we will just offer coffee. We will dive into the data set to show which data we have, how can be used and to get feedback if you are interested, et cetera. So this is all. Now I left some time for question and answer because I was getting more in what you are interested on. This is the website. Here are the two links to GitHub. But that's it. Thank you Claudio. So there are two microphones. So if you have questions, please come here and share them. First question? Well, first off, thank you for doing this because I think these things are important to open our eyes to give us an eye into this kind of stuff. I'm wondering how big are your plans? Are you looking also to integrate that across multiple social media? Are you going to integrate Twitter, Google News and stuff like that? Maybe. I'm open to that analyzing the impact of the social media is important too. But Facebook is quite meaningful now for the importance that it's having and this is the first practical way to test the community, to test how much is the answer in the interest, to test how much we are able to manage the supporter and how much are the problem in managing this data because starting now to support others can be just a dispersion of effort but I hope in the future it will happen. The nice aspect of tracking exposed is that it's a domain name who contain a message. So you can put all the subjects you want as a level. Next question? You did this experiment with both users, this had slightly different likes. Did you do an experiment like have 12 or something, users having exactly the same, seeing if they get exactly the same results or if Facebook is adding some element of random to this? One theory is that Facebook will put an element of random. The other theory is that in the moment you are registering a fake user at two different times. When you put the likes, you're just changing the graph somehow. And that means that the next user will act in the graph in a different condition. Or it can be random or it can be so thin that I don't think we can really measure it. But we also have not made a test with a completely equal user. Our tests were equally enough to do this table but also, for example, what has got influence was the size of the screen when you're doing automatic scrolling. So the kind of element that you discover during the test that can lead to use some differentiation are many. All the data points that Facebook has are just unimaginable. But if you want to run this test, there is all the tool for you. Thank you. Next question, please. Hello. Thank you for the presentation. Do you obtain data about the publicity on the Facebook profile? On the privacy? Or how much? Publicity. Publicity. Yeah, we're collecting also the advertising data. So you can, from your own personal page, you can download as CSV format all of your feed or all the promoted, sorry, sponsored posts that has appeared. And actually to the sponsored, if you look in Facebook, if we go in parsers, those are the parsers to extract the metadata. And promoted is about the sponsored post in which we are getting info, links, title. And the other feed are about the feed. So there are also a different set of parsers because the structure is a bit different. Okay, next question here in front. They are accusing Facebook of having implemented filters in the algorithms so people get lead into bubbles. And have you found that these bubbles are really now completely invisible to each other? Are they isolated? Do they fragment society into smaller groups which do not meet each other anymore? Thank you. This is actually one of the example of the social political problem. What I'm looking for. As we made in the last five years, many research. This is one that can be interesting for you. It's some other example has been made as a personal analysis. Now this one was in Ferguson. This one about the conservative vision. But this is a research made across many user. The bubble is a psychological and a technological phenomenon. So we as human stay with the person that we know. We are okay with, we share the same idea, we share the same language or values. That is physiological. The social media can be a problem when this kind of feedback that we as human tend to pursue is incentivated by the algorithm. So if the social media became an experience that can open you a little more than in the physical world is an achievement about openness. In the moment the algorithm start to prioritize your own friends that's became a lose. But this means that social media are still potential. The filter bubble is not just technical and understanding these three shoulders can be important to understand how much the social media has an impact on it. For sure is happen, yes, but also is complicated. So if you compare a person that was spending their own life without the social media and get connected, for sure the bubble is bigger. But if you're taking a person who has a certain lifestyle maybe traveling between cities, going to university and having multiple friends, the exposure to the different news maybe is reduced. So also stating a definitive way about the filter bubble is complicated. Ellie Parise start to talk about it six years ago and now a new research say, yes, I exist, but. So please, you are here in the talk with me. You have listened to the talk, do you have some question? Not really. Ah, okay. One question, can I use this piece of software to try to run my own algorithm for the news like making not a daily, maybe a weekly newspaper of my Facebook? Not the extension, the extension is just collecting. No, no, no, the API. Yes, with the API you can retrieve what you saw and have your own algorithm that sort out your alternative version. So it can be a way to experiment a different user experience of your Facebook data, yes. Thank you. I will love it. Next question from the other. I'm curious as follow up to the previous question there, how disparate the profiles could get, how the different feeds could get. So what is the maximum percentage of difference you found between profiles? Like what are the most extreme differences that you found? To be honest, I don't wanna do analysis. I want to enable other to do analysis because one of the thing that is happening now is that in a political reason, after the Brexit and Donald Trump event that has make everybody surprised, as because the media have to understand better how the social media works, there are many groups that are trying to do experiment in analyzing and providing research output. And this is a cross of skill. The skill of the team we are building is technical because we have to deal with the technicalities and enable other researcher to do their own analysis. The analysis I made has been just for the presentation to show there is potential, but I've not actually start to make these analysis because I'm not a social scientist. I can guess which are the difference between profile with the numbers, but then... I still be interested in your guess. I mean, just your general feeling. Were you surprised at how different it could get or it was not so bad? I have not looked at it. Really, also in the user I'm using, I tend to follow different source. Instead of this answer, can I work if you are following the same between friends? And in the experiment we made, they were the same forcefully. So I really not an answer. But it's not so difficult to do this test. Tomorrow, 10 to 11. I'll be there. Thank you. Next question. I may have missed this in the beginning, but did you share this information with Facebook and specifically in light of your recommendations, I wanted to know how open they are if you have shared them? No, I consider our group still in these sides of potential. No, these sides, the potential is bigger. I hope. But no, I've not got in touch with them. Yeah, thank you. Already. Doesn't look like that they are currently up for the questions. Claudia Augusti with the Quest for Algorithm Diversity. Thank you.


For 28 years, Fishing Has No Boundaries has provided its participants with memories that last a lifetime. In this week's Northwoods Adventure, our Josh Peterson shows us how devoted volunteers make it happen. For many, it's become a tradition that the final weekend in June becomes a chance to create some lifelong memories and of course, some great fish stories. Fishing Has No Boundaries has been proud to provide that opportunity to those who normally may not get a chance to experience the sport. We adapt to you. We make it so it happens for you. You don't adapt to us, we'll adapt to you. We'll do what we have to do to make things work. The success of the program in Bemidji called for an expansion of the program in other communities, providing even more people with a chance to create those special memories. There's an event in Brainerd that we helped start and there's an event in Devil's Lake that we helped start.  But what makes the event work is the support of all the volunteers. During the event, 22 boats were donated along with boat captains who donated their time over the weekend to make sure everyone could go fishing. Looking at the clients when they come in off the lake with their fish, they are so excited and to tell you about what their day of fishing was, I mean, that just, that kills my heart because that's what brought me back. For some of the volunteers, what makes it worthwhile are all the smiles from the participants. Whether it's a big fish or a small one, the experience is what counts. I re-earned and I gave it to the net and took pictures. With a day and a half worth of fishing, the experience for these participants created stories that will last a lifetime. With this week's Northwoods Adventure in Bemidji, Josh Peterson, Lakeland News. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


here. All right, thank you all. So, like I said, I've been doing open source for a while and one thing that I've learned over the years is that when you talk to your management about open source, they know that you're passionate about it, but they don't know what you're talking about because your priorities are very different from theirs. So, yeah, I have been working with the Apache Software Foundation since some year that started in 19, and I started out that way. You're extremely passionate about open source, and then you go and you talk to your manager about it and they say, like, you know, and how many of you all work with open source at work? So, you want to keep doing that. That's why you want to learn these things. You want to persuade your manager that this is a worthwhile thing for you to spend your time on. So, let's start with a question. Why do you do open source? Somebody tell me why you're interested in open source. Speak up. Scratching an itch. Alright, scratching an itch, implementing a feature that you want. What else have we got? Make a lot of friends. That's something that's not important to your manager. And in fact, you know, some of the things that we hear scratching your own itch, it's fun, socialization, altruism, giving back, making the world a better place, building your resume. These are some of the top reasons that people give for participating in open source. And your manager doesn't care about most of these things. And, you know, what what we love about post Asia is that we get to sit around a campfire with the people that share our interests and we get to talk about these things that excite us to talk about the technology that we're interested in. We all have funny stories about our management. We just doesn't get it and we like to laugh about it behind their backs. But next week, you have to go back to work and you have to tell them that your time here was worthwhile. And all of those reasons that people give for participating in open source, that is not why your company cares about it. Your company cares about making a profit. They care about serving their customers and making their customers happy. They care about making a profit. They care about retaining employees and they care about making a profit. And so see just most of the things that you want to persuade your manager about. Now, of course, I'm not talking about your manager. Your manager let you come here. And so obviously they're awesome. My manager is the president of the Apache Software Foundation. So when I talk to him about open source, he gets it. He understands it. But his manager is a brilliant person who knows a lot about product marketing, not a lot about open source. So when we talk with her, we need to make sure that we are communicating. Now, as I was writing this talk, it started coming across as how do you lie about what you do in order to persuade your manager that really you're working for them. That's not what I'm talking about at all. Because open source is primarily about pragmatism. It's not solving problems in ways that work and collaborating with people in ways that work. But the way that we talk about it tends not to communicate that. We talk about how fun it is and how we like hanging out with our friends, which is true. But again, not what they care about. Now, the other thing that I want to mention is that different open source projects are different. They're weird. They each have their own unique culture. And some of them have unpleasant people as maintainers. And so maybe the things that you try to do in open source won't be successful. So don't think that the things that I'm saying here are in promise. It's just sort of that you need to speak the language of management. This is about translation, translating the things that you're passionate about into things that your management will actually care about. And like I said before, it's important that these are things that are true. You know, we're not trying to we're not trying to pull the wool over management's eyes so that we can go and play with our friends. So remember that open source is above all practical. It's objectively a better way to build software. And this is supported by study after study. Open source software has numerous advantages. We just need to talk about it in ways that convince our management of that. And in order to do that, you need to think about what's good for the company. What's good for your customers. And so, you know, I work at AWS, one of our catchphrases is that we are aggressively customer focused. Everything that we do is about doing what is good for the customer. And so you need to know who your customer is, right? Well, like when you're giving a conference presentation, you need to know who your audience is. When you're building a product, you need to know who your customer is. So you can't just go and hide in your office and write code. You have to understand what your customer is. You have to understand what you're delivering to them. And that's part of what's in it for the company. Another thing that's in it for the company is the cost. What does it cost to develop software? Does open source help that? Maybe it does. It depends upon what particular part of the industry you're in. Another thing that open source gives to your company is reputation. And so that's one of the other things we'll talk about. Now, rule number one when you're talking to your manager about open source is don't get bogged down in philosophy. They don't care about the distinction between free software and open source software. And you and I could talk about that for two hours, and just scratch the surface of it. Your manager does not care.  So another thing that they don't care about is license. They don't care whether your software is Apache or MIT or AGPL 2, 3, or 4. It doesn't matter. What they care about is whether it serves the needs of the customer. The other thing to remember is that you have a minute. You don't have two hours. You don't have the opportunity to say, what you really need to do is go read Eric Brayman's Cathedral in the Bazaar and then get back to you. You have a minute. You have the time between two floors right in the elevator. And if you cannot communicate during that time, then you have missed your opportunity. I remember one of the earliest open source conferences I went to. Richard Stahlman and Miguel de Acasa were discussing the relative benefits of open source versus free software. It was just high glazing. It was there was no point that came out of it. Except when Miguel said, if we have this conversation in front of our management, they're going to fire us. What we need to focus on is the solutions, the problems that we're solving. So so what are you going to talk to them about? Some of you all said some of the people in the survey said that they're interested in giving back to the community. Your company is not a charity. They are not interested in giving back to the community. Management sees open source typically as a renewable resource. It's this free thing out there. We go at harvest. And then we build our product with the harvest.  If you don't tie it to a business benefit. So giving back the way that you should instead talk about this is the supply chain. Your manager read a book last week about the supply chain and software bills, bills of material. And they're they're anxious to talk about that, because that's actually relevant to their business. Now, if you are building a product on top of an open source project, that is in your business's best interests, that that project is healthy and sustainable. So if that project is hosted at the Apache Software Foundation, maybe you should consider sponsoring the Apache Software Foundation. If that project is at the CNCF, maybe you should consider having one of your engineers working full time on that upstream project. These are ways that you can sustain the project so that your business is around next to you. It's not just about downloading the version that was released yesterday. It's about ensuring that there will be a version released next to you. And so talking about getting back to the community, it sounds very philosophical, talking about sustaining the supply chain, so that that little red paper doesn't break next week. That's that's what's of more interest to your manager. obligatory xkcd and you've seen three times already this week. If you are working on this project here and it fails and your business is a thing about up on top, you might be out of luck. support these arguments with data. If your product is based on open source, then you can say 73% of our profit is directly related to the success of this project that you haven't heard of until last week. But if we don't contribute our fixes back upstream, then that portion of our profit is at risk. So this is a quote here from that's actually not a quote. It's a made up press release. But if you can say something like this, Apache Commons is a critical component of our product. And that product earned us this much profit last year. And if that project fails, our company fails, data is your thread. Sustainable open source is a project that's around next year. You have to think long term. So things that you need to think about in sustainable open sources, are we building this project? Are we building our product on top of a project that is developed by our competitor? Is that a problem? Maybe we should go be more participating more in that project so that it's not a single vendor project. Are we building our product on top of a project that is maintained by one engineer who lives in the Czech Republic and is unemployed right now? Maybe we should hire that guy. And then also maybe we should devote an engineer to that project. So that it's not just that one person, but it's two or three people that are supporting this so that maybe that engineer can take a vacation and not burn out quit next month. If we participate in this project, then we are by proxy making our customers heard in the roadmap for that project. That sounds important. These are all aspects of sustainable open source projects. To make sure that your stakeholders are heard when they're discussing what's going to go into the next version, rather than just waiting around and hoping that maybe that next version will fix this bug that we care about, because it won't if you don't participate. All right, another reason that people give for participating in open source is earning merit and reputation. And I'm probably the only person in this room old enough to have seen that movie. So maybe that's not a good screen there. But it's from a movie called The Breakfast Club in the 80s, where these two individuals were vying for reputation in school. But your manager doesn't really care about your reputation in open source. So instead, talk about earning trust and influence in the project. So for example, if you are building a cloud based solution based on top of Kubernetes, and there's a bug that's affecting your customers, and you go and complain on the mailing list, you're going to get ignored if they have no reason to listen to you. So if you participate in that open source project, if you have an engineer in your company that is contributing actively to that project, then your company will buy by osmosis gain a reputation within that project. And they'll be more likely to listen to you when you complain about a particular bug. So this is a way to convince your your manager that participating in that project is actually worth their time and their money to invest in that. You should be careful when you participate when you try to participate in open source, that you actually understand open source culture. It can be dangerous to talk about how your project, how your product is built on top of an open source project. And then the people in that project say, this is not somebody that participates. This is not somebody that contributes. They're just they're just benefiting from our work. And you know, this is a problem with many big companies, not naming any in particular, that build their products on top of open source software, and then go out and talk about how they're an expert in this field. And then the community comes back and says, no, they're really, they're really not. So you need to be careful that you what one of the phrases that I use at work is be don't seem, which means don't pretend that that you're that you're great in a particular open source field actually be that way. Go do the work put in the time become an expert, contribute and earn that reputation be don't seem. Also talk about adoption. Slide in my on I need to pick up the things just a little bit. Talk about how the adoption of an open source project benefits your company that's based on top of that open source project. If you have expertise. So what we found at AWS is that people don't choose AWS. What they do is they choose a technology. And then they go and they look to see who's the best at running that technology. Is it Google Cloud? Is it Azure? Is it AWS? I want to run Kafka. What's the best place to run Kafka? Is it Confluence? Is it AWS? Let's look and see what the reviews say. But first I've chosen Kafka. And so you need to talk about how participating in open source builds that reputation, which then drives adoption. Alright, here's a fun one. We love to talk about collaboration. I work at Amazon. And if I work on this project, I'm going to be collaborating with Google. And management says wild earth would we want to benefit our competitor? So what your manager hears is that you're going to lose control of your product by seeding that control over to a competitor. Instead, talk about focusing on what your company excels at. So again, I work for AWS what we excel at is infrastructure and scale. You know, we have some people that are working on Kafka and they're they're good at it. But that's not what we sell. What we sell is the man chose it, right? So talk about and so making commoditizing the software allows customers to build and test home on the software that they can download and use for free. But what we're selling is not the software what we're selling the service. So this this notion of loss of control is is a bit of a sidetrack because it's not about controlling the software. It's about influencing where it goes based on your customers. But everybody's customers have similar issues that they're trying to solve. And that's not what we're doing. All right, some of you all said that you do open source because it's fun. Open source is an endless party. I get to travel around the world and hang out with my best friends. And I've been doing that for 20 years. And it's awesome. And I'm very careful not to say that to my manager. Instead, talk about recruitment. Talk about how participating in open source makes your office a great place to work. And that makes it easier to acquire the best employees. Your company doesn't mind you having fun, but it's not really what they're there for. Now, the flip side of this is make sure that if you hire someone with the promise that they will get to work on open source, that they get to work on open source. Because if you hire someone and then retake, hire someone and say, you're going to get to work on a patch of airflow all day every day. And then when they get there, you say, but you know, actually, what I need you to do is spend the next six weeks solving this JavaScript problem over here. They're going to tell all their friends, and it's going to be harder to recruit people. So make sure that you make good on your promises. Note that open source people, whatever that means, can be very opinionated. They tend to lean towards being archaic and libertarian. It can be difficult to manage. They want to go and do their own thing. So make sure you understand what you're getting into when you hire open source people. But they're also very creative. I mean, look around you with the brilliant people in this room. It's really a great opportunity. If you let people work on open source, it feeds that creativity. It gets the best of the world's engineers to work on your product. You know, one cool thing about open source is that no matter what company you work for, the smartest people in the world work somewhere else. And if you can get them to collaborate on your product, then you're winning and they're winning. Another thing that people often say open source is good for is building your resume. Your manager is particularly not interested in you building your resume. Instead, talk about continuing education. I can go work with some of the best engineers in the world on a piece of software. I can learn from them and someone else is paying them. This is really, this is a really win-win situation. Now, one of the other things that your manager might think is that open source is free. And I like to say it's free as in free puppies. When somebody gives your kid a free puppy, that is not free. Be very cautious about talking about open source as free or cheap or reducing costs. You know, it might, but those costs go somewhere else. The software is free. You start to hire people to maintain it and to build it. So instead of talking about customer value, open source builds better software. Software is easy. People are hard and you want to ensure that you are investing your people in the right kind of development. And I am almost out of time.  Because that means that they have to maintain it forever. So make sure that your management understands the right way to interact with open source. And that's basically my day job is helping management understand the right way to interact with open source. Let's see. Yeah, I'm out of time. Does anybody have any questions? I can talk about this literally for hours, but does anyone have any questions in my last one minute? Yes. How do I have the conversation about getting them to open source something that I've created for internal use? So the question is, how do you get your management to agree to open source something you built internally? And it's really hard. You have to be able to demonstrate. So first of all, you need to know why you're open sourcing it. And you need to have an answer to that question. And that's going to be different for every project. And there are many reasons to open source something. The other thing you need to be able to convince your management of is that you're not giving away the secret sauce. And in order to do that, you need to understand what your business is. And so this is what I was talking about earlier. AWS doesn't sell open source software. We sell the services on top of that. And so you need to be able to clearly communicate that the software is not the value and that the collaboration is only going to enhance that. But yeah, it's a hard conversation and everyone's different. You need to make sure you know your own reasons first. There is someone over here. Yes. Yeah, it's just values now for, for, I would say software, but the, the fact that you cannot do that.  Because now, like, oh, why she's one of all that. Yeah, that's cool. Yeah. That is a very large question that we don't have time for. But yeah, I mean, you know, again, it's, it's always going to be very, very dependent on a specific type of project that you're working on. But I don't know, maybe take that after this is completely out of time. Any suggestions for the managers? So what I would suggest for the managers is that they attend this conference. I mean that very seriously that they attend this conference or if they're in Europe, that they attend FOSDA or if they're in North America that they attend the open source, the Linux foundation's open source summit. And the reason for that is that open source is 75% culture and, you know, 25% software and understanding the culture and the collaboration behind this, understanding that when you get into a room with people like this, you hear weird presentations like the one that you gave that open your eyes to something that you hadn't really thought about before. That's, that's really, you know, that's not something that I would have studied on my own, but attending your talk, my eyes were open to something that I had sprang in, right? And so attending events like this, I think is the answer. Thank you so much.


I don't know if I've had many years of experience, and I don't think yours actually teach you anything, but thank you. So what are we going to talk about today? Like what I was saying earlier, at PayPal we have several use cases where we deal with massive amounts of data, some cases streaming, some cases analytics, some cases are mixable. So what I'm going to cover today is one use case for site monitoring. So if I think a mature company like PayPal, opinion business is very successful, we would have figured these things out day one. But we're the world's biggest startup.  First it's about making money.  So things like how will this thing actually stay available, how will it scale, how do we manage the system, how do we operate it, take a little less importance at the onset. And if you have success like PayPal, well, it comes a little later than normal. So when I joined PayPal a couple of years ago, I was told, well, we need to figure out sort of how we improve our monitoring capability. And when I looked at it, our monitoring system started from application to server to the switches. The entire stack was generating about 250 billion events in a day and about 20 terabytes of data. I didn't try to correct me if I'm wrong, but now our application logging is at that level. So this has grown to probably about 350 billion by now. And it's growing on a daily basis. So how do you make sense out of that bigger stream of data? What do you want to do with it? When you talk about monitoring, that's a pretty big sort of wide open scope, right? What is it that we really want to know? So we say, okay, there's two more things we're going to focus on. We're going to try to do real-time analytics. We're going to try to do some basic correlation. And I hope you guys all know what correlation is, right? So if you have an event stream trying to figure out which events are really related to each other, even though they may be coming from different parts of the system or different levels in the stack, detection is actually a big part of monitoring, right? How do you actually know some things that are wrong? And then of course such, right? So monitoring applications, generate passive parts of logs, trying to actually go through those logs and find the little bit of information that you're actually interested in. So of course the first inclusion everybody's out to. Let's put all this data in a database, right? And then we can, like, query, right? And then we can figure out what went wrong. Easy. Everybody knows SQL, right? All of my interviews are about to change. I said, well, you know, it doesn't sound right, right? 250 billion events, 20 kind of bites a day. What else is out there? Everybody said, yeah, let's use Hulu. Great. Okay. But when we actually did sort of even a basic analysis and a look of concept, right? I had a budget of $500,000. And to stick this thing in Oracle would cost me $600,000,000 just in hardware. Right? Scalability is a concern. Traditional IDPMSs are scalable vertically, right? And as you scale vertically, the cost of that server goes up and up and up. Right? Okay, so Hadoop was supposed to solve that scalability problem, right? So yeah, we can get some commodity servers spread from across and that will work. Well, what we found out was it takes about 30 seconds to start the quit. Obviously not going to work, right? I want real-time analytics if I find out. So don't get me wrong, we're not at 30 seconds even now, right? But we're getting there. But with Hadoop, it wouldn't have been possible to get past the 30 seconds. So what do we do then?  And about 2,000 simultaneous queries on this data. Right? That was sort of the benchmark we were talking about. We said, okay, let's look at, you know, what's intended for sort of real-time analytics and other things. Columner databases was one of the things that came up, right? So we looked at vector, sort of vertical, vector-wise, and part-self. So columnar databases, of course, I don't know if you guys are familiar with these, but instead of trying to store data in rows, they tend to scale. And a simplistic explanation, store data in column structures where columns, sets of columns actually then make up a table. Right? Or a data grid. You get spaces and some appliances which were data. Both of these work very well in terms of actually performance and in terms of giving us that sort of capability to do real-time analytics and scale it out as time went. No promise has the lowest cost solution for hardware, software, and power. And they're designed specifically for real-time analytics use cases for streaming. A lot of companies are using them for this. They scale linearly. Since they are sort of linearly horizontally splittable, the availability factor would be had an availability requirement of 4 nights. 99.99, right? They meet that requirement. So the number of nodes you can put in, you'll meet the availability requirement. But again, do we have the money? Right? All of these things come at a cost. And I think that's how solutions like, you know, Hadoop and any other workforce first came apart, right? Someone's part of their way to solve a problem, solve it at low cost, solve it efficiently, solve it for all the parameters that are important. So these solutions, although they were suited for what we were trying to do, had a cost associated with it that we couldn't bear. For example, just as cost. So if you get spaces sort of linearly, we would need 500 nodes just to set up the data. 500 machines, right? So you can imagine, you know, a whole data center today runs about 2,000 machines. So I would multiply the data center footprint by, you know, 25% to just get sort of visibility into the data. So what we ended up with was sort of a custom multi-dimensional aggregation mechanism which was not invented by us at PayPal, but eBay had this, you know, working for about two years. And as part of the analysis, and this is proprietary, so I can't tell you the details otherwise I'll have to knock you in the room forever, right? So this is what we end up using, end up picking to actually go implement on our side. There's an ad-hoc query system for read-only nested data. This is sort of the biggest problematic query when you actually talk about real-time analytics. We get meaningful compression out of it by actually aggregating data and not trying to store all of the raw data. And then the time series data, right? So split off different types of data into different data sources. So all of the time series data now goes into a run-up in data history. So the first thing I do is it's really synonymous with big data from all the sort of things that we played with. But it does work for simple batch processes, right? That don't have hard real-time requirements. It's pretty efficient, you know. And now things like iterative processing are being added to Hadoop in the recent leases, and when they go to sort of one dot or one things like that. But many tasks, things like anything to do with absent requirements, right? Anything that requires asset properties. Anything that requires continuous incremental updates. Hadoop is not a good fit there. And those are the use cases we have. So someone earlier asked a question about the actual transaction process, right? We use OLTP systems for transaction processes. We use Oracle at the scale that we run at. At the size we run at. We still use Oracle because an RDVMS is even now best suited for transactional type of use cases. So the lessons learned out of this, there is no one solution. There's certainly a way to jump to conclusions and try to fit one solution into the problem space you have, but there isn't one solution there. You have to pick what fits you best. And that same thing goes for big data, right? That's all of the general principle I try to follow. I think the same thing fits one of the big data. So I guess, do we have any questions? I know we have 10 minutes to get to that. I hope I didn't use up anyone else's time. So any questions? My name is Chandu Nayak. There's a lot of things that you've talked about seem to be largely structured data. But a company like Vepal and Swarman are giving a lot of unstructured data as well. How do you handle that? Number one and number two, how do you integrate it at all with structured data? So you're right, a lot of the data that goes into things like logs and other things is unstructured. But again, if our primary use cases for analytics are based on at least semi-structured data. So outside of log files, we don't have, and even for why the centralized application logging is in place is to put some structure on the data so that it's not completely unstructured. So the way we have it is it's a constrained set of fields. And yes, there's variance in the values that goes in these fields. But it's not completely unstructured. We don't have major use cases where we view in sort of documents. Even our transactions, so once the transaction actually is paid by, again, it's constrained by the interfaces we provide. It's sort of structured by what data we actually expect in a particular interface. And same goes for the interchange of the partners. So most of our data is at least semi-structured. And we don't try to sort of marry it with structured data, like I said. We try to split it not on the data here, but more so at the application. And I have one question. When you said about real-time analytics here, what precisely, I mean if you could tell that, what precisely is that real-time analytics, especially with PayPal? So for example, one of the prime use cases is a decline. So when someone submits a transaction, that transaction can get declined for a variety of reasons. Your credit card was expired. So we detected that this is a fraudulent transaction. It's coming from an IP address in a geographic region which is not the same as the geographic region you normally do a transaction from. And many parts of the system can actually decline the transaction. And external partners can decline the transaction. So decline is a business metric we track very closely. So at real-time, we want to know how many declines are happening from which part of the system or from an external network. Another example, we have cost associated with processing. In many cases, in some countries that we go to, our transaction volume is a fraction of a percentage compared to the volume in the US. So if we have an outage for customers in that country, it will not be detected in the overall health of the system. But for that country, for that processor, it's important for us to know if the transactions are going through or not. So to figure out, in a country like France with a particular processor who charges us less than another one, are all the transactions going through as they normally do? That's not a case of real transaction. So it's more O&P than a prediction kind of... No, it's not transactional, right? So it is analytical. Yes, probably O&P. It's saying that you just pull the data out of the warehouse every single minute or every single second. You can't pull that much data out of the warehouse and you can't feed it to the warehouse. Yes, but yes. It's processed into streaming there, right? That becomes important. So, are you doing it? Besides... Like, since the other day we were doing this data, what are the possibilities that we're going to have? The client is a dude. How many are genuine and how many are fake? Can I lock you in a room? Sir, we're not telling you how we're doing it. No, no, no. So there's a percentage of declines that we do that are false positive and that are sort of positive, right? That is a business metric that is not publicly sort of available. Because that's... We don't even expose our risk rules or anything related to those rules even for task purposes to our external partners. So if you're trying to test against our system, you don't ever hit a decline, right? So there is no way to actually yield. Percentages are there. We measure those entirely and they're under 5%. I will not give you the exact percentage, but there's under 5% that are sort of false positive and that's where sort of the metric is tracked for our risk systems to actually track continuously lower. Sir, why are you taking this... The recent experience is there for me. How many? Two of my genuine transactions declined. That's what we can tell you, right? So can you start coming out and start running? So one solution I have is that, you know, each speaker can actually take a couple of questions but then we're going to open it up for questions and ask a lot of questions. I mean, it may not be about your transactions with PayPal, preferably big day and hour later. So we'll be glad to help you and sort of tell you why your transactions are actually declined. If you can give me a PayPal account, I need a password or anything. I need something to start the investment.


And to Murray, who I'm sure is watching us today from our pie, I say, thank you Murray. You are my hero. I shall not look upon his light again. And I hope you are happy with your student. I always felt tremendous joy when you told me, great hunts, other boy, and even if I can hear you right now, nothing would give me greater pleasure than if you said it again right now, up there where the kings of thought are gathered.