Welcome to Economics and Beyond. I'm Rob Johnson, President of the Institute for New Economic Thinking. I'm here today with Arjun Jyadev, who I'm pleased to say is a senior economist at INET and a professor of economics at the Arjun Prenji University in Bangalore, India. We're also here with Achal Prabhala, who is a fellow at the Shuttleworth Foundation and the head of the AXIS IBSA project there in India. Thank you both gentlemen. I've read your articles in mint, live mint. I've read your articles in the project syndicate. I feel like you're dealing with very, very important problems and my audience and I are very, very curious about your insights regarding India. You're both based in India and I thank you for joining us. We need to shed light on all of the places that are important in the world. Thank you, Rob. Thank you, Rob. So here we are at the end of May, 28th of May. We have this global pandemic. It is hard to be jovial or humorous about such a thing. But I do make the joke that precisely at the time that all humans are asked to wear a mask. We are unmasking the flaws in the economic and social organization paradigm that we've lived by for the last 40 years. How do each of you experience this violent and dreadful tragedy? And what do you see? What do you see that is being done well that gives you encouragement? What do you see as missing that you would recommend? What creates fear within your spirit when you see dreadful things taking place? So I'll ask you, let's start with Arjun. Sure. How do you see the pandemic and the challenge that it presents? Thanks, Rob. I like the phrasing unmasking of the paradigms that we've lived in. They say that this particular virus exposes the weaknesses that already exist in the human body. But I think that through as much of the body politic. And I think across the world we are seeing, you know, tearing along familiar fault lines. And, you know, my experience in India has been that this has really exposed some of the really horrendous fault lines that we've seen in India, specifically on the way that we have dual tracks of life and death in this country. I suppose this is the case all over the world, but the way that we have our responses and our general sense of responsibility, civic duty seem to be completely divergent. And the pandemic by virtue of making sure that everybody is under threat has exposed really this underbelly. So just for example, we're now in India in the 68th day of lockdown and we've seen possibly the most horrendous internal migration that has been there since we had partition. And day after day we see horror stories about people having to walk home, whereas and walk home meaning 1200 kilometers they given trains trains get lost. It's, it's both a huge amount of malevolence and incompetence when it comes to the poor, the rich are hiding for the moment, and we can't hide very much longer. So we're going to see that play out in the next few weeks and months. So I think that there are many, many things that we can explore further but just by way of putting the broad canvas I think it has exposed these dual track and our response thus far has not been promising with respect to handling this in a collective way which really we have to be. Charles, how about yourself? How do you see what's on the horizon? Where do we have to address things and where are we lacking? Thank you, Rob. Much like Arjun, I think I live in India. We both live in Bangalore in South India, which is a relatively prosperous city with relatively young demographic. The thing that's been possibly the most heartbreaking in the last three months has been to see not unfortunately what's happening directly around the coronavirus pandemic, but to see the thousands, hundreds of thousands of people who have no safety net whatsoever, who are completely at the mercy of a daily wage, who are being denied that daily wage, who are being denied food, who are being denied basic shelter in the places that they work in making these tracks back home, starving along the way, some of them dying along the way. It's impossible to watch without bursting into tears. It is a horrendous strategy and what it underscores is the callousness of the Indian state. We do have welfare programs. The BJP, the party that's in power in India, despite being a politically right-wing party, cannot undo the welfare schemes that were put in place prior to their rule. And so we have a few, not enough welfare schemes in place around basic employment, around food security. Very few of those are accessible to migrant workers when they're away from their villages, their homes, their states. And it has exposed the absurd callousness by which the Indian state machinery treats people who are literally at the bottom of the economic ladder in this country. And I think that's been extremely hard to watch. But what's also been hard to watch is how the BJP and the ruling party has brought every preexisting prejudice that it has had over the last several decades of its existence. Somehow into the pandemic. So very early on in the pandemic, the BJP targeted a group of Muslim scholars who met in Delhi and identified and then shamed them publicly as being somehow creating a super spreader event for COVID-19. And that quickly turned into the country's preexisting prejudices and turned those prejudices on, which meant that in the first few weeks of our response to the pandemic, the country was mainly interested in blaming a very small group of Muslim scholars who were among several other people who had been inadvertently caught up in the mass chaos that the pandemic response created. And so I'm almost stupefied, but almost admiring of how the BJP has managed to bring every fault of its from the past few decades into this pandemic, completely distracting us from the essential work that needs to be done, not just to crest the wave of the coronavirus, but to address all of the underlying problems that is exposed in the way that we treat the most vulnerable people in this country. So it's been difficult to watch. Arjun, the two of you wrote an article recently about what you might call the dilemmas might be a nice way of saying it, but the potential damage of allowing intellectual property rights in the pharmaceutical industry to prevent a broad based response to the pandemic providing vaccines and other medicines. You compare that to the flu vaccine and to Jonas Salk and describe to me and to our audience what what is it that you are are seeing as we might call the failing of the current system. Thanks Rob. Yes, so just as background to to this. So Achal and I have known each other for for decades and we went our own separate ways and did many things and I did economics. Achal did law and many years ago we came back and found we had a common interest in trying to understand how we should think about this intellectual property system, specifically with regard to the question of medicines and we've been working on it on and off ever since sometimes with Joe Stiglitz, who is, you know, my postdoc advisor and so what by way of background, I think it's important to to understand that the intellectual property regime is not sort of natural God given thing but something that was created explicitly in the last 20, 30 years in order to push particular advantages that richer countries had specifically within the pharmaceutical industry. And it's been an extremely destructive system globally in terms of purely of lives, but also in terms of destroying the country's indigenous capacity to respond to public health emergencies and so on. India has a particular interesting history in all of this because India in 1970 actually banned any intellectual property rights and medicines. And as a result, we had an indigenous industry of generic medicines which has its own quite incredible history, because of the country's generic industry, southern African countries and South Africa in particular, and Achal will probably have a lot more to say about this was able to benefit from much lower prices so we've been spending a lot of time on this but stepping back a bit Rob I think the really interesting thing is that throughout the history of public health and medicine, there have been many many innovations and many many things that people have done without the the requirement for and use of intellectual property. And of course the classic example is Jonas Salk, but they've been many many things that people have done, which have been supported publicly through things like the National Institute of Health or through universities, which have then been taken up by private companies and been and the common the knowledge commons have have have therefore been enclosed, and this has become an extremely destructive system globally. And so we've been writing many many responses to this including what you can do differently and we're not of course the only ones have been many kind of thoughts around this advanced market commitments, prizes over patents, making clinical trials publicly funded, for example Dean Baker has written a lot about these kinds of things. But I think what is interesting for us as economists and lawyers is that we have all these interesting ways in which people are actually doing innovative things, which does not involve patents and intellectual property rights, and that we ought to be supporting these kinds of endeavors far more than we are. And that was the motivation for writing much of this much of may have something more to say about that. Yeah, no certainly I do. Look, I think I've worked as an activist now for about 18 years and I've been working on access to medicines since then I began getting interested in it at exactly the time that the AIDS crisis in the developing world spurred and actually created the access to medicines movement. And so at the turn of the century, or the millennium, there was the confluence of two things that that created both tragic circumstances as well as a wonderful activism that has had far reaching effects into the present day, which was the creation of the WTO in the late 1990s, and a certain formalization of a global intellectual property rights that were tied to the trade regime and trading rules and therefore given a set of teeth that that bit much harder than that ever done before. But that confluence consisted of two things, not just the globalization of trade, but also the emergence of countries in South Africa, for instance, especially South Africa, as being the global centers of the HIV AIDS epidemic. We thought of that as being something that was restricted to Europe and North America in the 1980s because of what we heard. But by the 1990s it had shifted to countries that were far poorer. And what they found was they were in an acute crisis of affordability. This was a very strange situation. There were cures in the sense that there were drugs to treat HIV AIDS, to turn it from a death sentence into a chronic condition like diabetes. So something unremarkable, you could, you know, from being the cause of so much hysteria, HIV and AIDS had suddenly become, had the potential to become something unremarkable, except that it couldn't quite be that way in South Africa because the drugs cost $10,000 a year at the time. And for poor countries with per capita incomes that were several times lower than that, it was obviously unaffordable, either for individuals or for the state to do anything with that solution. And this seemed particularly unjust and a particularly cruel metaphor for a kind of emerging neoliberal economics at that time that you had a cure you simply couldn't afford it. And that's how I started off doing this work. The AIDS activists of the time, by the way, did really create the template that then worked to solve access to medicines across a range of diseases from cancers to hepatitis C. It moved from being a third world activist cause into becoming a first world concern. I worked on a campaign in the UK to gain access to drugs for cystic fibrosis. During the Democratic primary campaign, there was not a day that went by when Bernie Sanders or Elizabeth Warren did not talk about measures to rein in drug pricing. And all of that actually the template for this activism and the spotlight on these unjustified and frequently just outrightly unjust private monopolies that pharmaceutical companies held on life saving medicines. The template for that activism really did come from South Africa in the early ups. And I think one of the things that's been welcome for me in the coronavirus pandemic is that in the past we used to think of access as an afterthought. The focus was always on getting a drug, a cure, a treatment or a vaccine out. And then the idea was let's let's think about how to get access to this vaccine for poor people. So with AIDS drugs, it took 10 years after the AIDS drugs were developed for them to reach the places that they were needed the most. We have cases with vaccines such as the pneumococcal conjugate vaccine, which is a very effective vaccine against pneumonia, which is seemingly taking an infinite time to get to India because we simply can't afford it even today. And so there are 127,000 deaths a year from a lack of vaccination to this completely preventable disease. So what's happening in this pandemic is that many of the people who are thinking about this, both at the WHO and elsewhere activists such as myself are thinking about this together. We're thinking about developing treatments and vaccines, as well as how to have access to them at the same time. And it's for a couple of reasons. I think one, there is this is universal in a way that AIDS was not or that pneumonia is not or that another major public health problem like tuberculosis is not. And it's forcing us to understand that the health of each of us is directly connected to the health of all of us and access therefore is an integral part of the solution. Even a nationalist solution necessarily needs to build in global access to whatever cures that emerge for COVID-19 in order for it to work. And so that's a welcome change. I think that what we're trying to do now is to see how we can effectively apply the lessons from 40 years of human history. Lessons that were hard and tragic battles that were hard fought and one on the bodies of people who succumb to these diseases. How we can apply those lessons in a way where we don't treat this as an anomaly. I think that there is a danger in treating the coronavirus pandemic as an anomaly to which the response is also an anomaly after which we then go back to doing things as usual. And I think that to the extent that I think there are two problems here. I think that the first problem is in terms of our access to medicines and vaccines to be able to carve out an effective solution to exit this pandemic wherever they emerge. But I think that the second problem here is to not treat this situation as anomalous and to treat it as part of a historical continuum, which we must learn from in order to change some of the structures we have to respond to pandemics like this so that we can have a different future rather than the same old past. Yeah, yeah. Well, I guess I want to, I think you guys have made a very clear argument and it's fascinating. Achal, you work in South Africa, India and Brazil from my understanding. Is that correct? I do. Yes, that's right. I work in all three countries. Yes. And in Brazil, my closest friend from my graduate school years and an INET board member is Arminio Fraga, the former central bank governor. And the sense that I get from he and his family members is that things are very, very dire in Brazil right now. This is a very, very daunting challenge to the fabric of their social organization. You know, I think that Brazil has had compounded tragedy upon tragedy over the last couple of years. They've had a very, very contested and difficult election. They had a whole series of austerity measures even prior to the election that hobbled the state's ability to respond to poverty and inequality in the country, which were, I do think, you know, however one evaluates the reign of the workers party, the pete, the key focus of pete for more than a decade from the late 1990s until 2013 or 14. And I think that this particular crisis, I've been writing a little bit around the Brazilian response to the coronavirus pandemic and I work with several colleagues there. I think that the human tragedy of this is unfolding in ways that we don't even understand when I was working on a couple of pieces about two weeks, two and a half weeks ago. The debt toll stood at 5,000 people and today, just in that very short space of time since we wrote the piece, the debt toll is over 25,000. The number of doctors infected is going up, the number of medical personnel infected and infected, mortally infected with the coronavirus is going up. The social fabric of the country is being torn apart, not in the least by the destructive encouragement of Jair Bolsonaro, the president of Brazil, for Brazilians to simply somehow ignore this pandemic and to sort of let it pass and to continue to indulge in his favorite pastime, which he sees as being key to his survival, which is economic activity, to keep the economy going at the cost of now it seems clearly thousands and thousands of lives. So I think that the tragedy is going to unfold in ways that we don't even understand. But I want to point out in the context of Brazil, something that I think people don't fully understand is that the classification of the country as a middle income as an upper middle income country in fact has hurt it immeasurably. When you look at the economic demographics of Brazil, Brazil is a country with a majority of poor people and a minority of rich people. And the truth is that for all kinds of purposes of charity and concessions made by pharmaceutical companies around the world, it's middle income countries, those countries that are unfortunately rife with income inequality, typically Latin American countries, some of some countries in the form of Soviet Union, North African countries that are in the worst position possible. So if I may, there's an example with one of the first treatments that was approved by the US Food and Drug Administration to treat the coronavirus pandemic, which is which is I think an illustrative example of how access initiatives fail and particularly failed countries like Brazil. So Remdesivir is an antiviral drug that was approved by the US FDA. It's not a hugely important drug in itself, but it's a drug that holds potential for further research as well as for use in combination with other drugs. And so there are multiple trials along the way. Remdesivir is the monopoly property of Gilead from political multinational giant, which has several patents in force for the next 20 years. Gilead announced an access agreement. They signed deals with five Indian companies to produce the drug and export it to 127 of the poorest countries in the world. And that got a lot of press. What didn't get a lot of press is that Gilead excludes about half the world of 4 billion people from its access agreement. Every rich country and every middle income country. So there's not a single country in Latin America that can get access to this drug at a discounted price. The Brazilian health system, which is trained, which has had its budgets frozen effectively for the last five or six years, has to face a bill of $4,500 per 10-day course of Remdesivir when and should it want to use this. While Gilead itself, without having sold yet a single pill, a single vial of Remdesivir, has seen its market capitalization increased by several billions of dollars in the last two months, while at the same time donating its entire stockpile of Remdesivir to the US government for use in the pandemic for free. So it's allowed the poorest countries in the world to make it with zero royalty. It is donating its stockpiles of Remdesivir to the richest country in the world for absolutely nothing, and it's charging an absurd price to every single country in between. Please. What a mess. And how would I say, those who choose to deify the marketplace had better start asking questions with outcomes like that. That's just not what you might call humanly justified in any way, shape, or form. Let me shift the focus now. You both live in India. And let's talk about, when I imagine a place like America using its fiscal capacity or Great Britain, there's just a tremendous, we might call, possibility or capability to respond to a disaster. In a place like India where you have high incidence of many diseases before and unrelated to the coronavirus and you have lots of poverty and you're at a different stage of development, the question I ask is, how can you mobilize the resources? How can you devote the resources? How can you acquire the resources to embrace this challenge as, how we say, given its severity and what is warranted for the humanity of India? So Rob, I think this is a great question. Just to link up to what we had spoken about before, I think it's an interesting thing that, you know, the three maybe outstanding authoritarian figures, Trump, Bolsonaro and more, have all been found wanting with respect to responses. And I'd like to say a little bit about India's overall response and the fiscal front. What really we've seen, another I'm asking here, is just a complete inadequacy of responses by the central government and it extends quite extensively for the fiscal response. About two weeks ago Modi was on television with one of his 8pm addresses, which of course fills most of us with fear because we have no idea what's going to happen. And he announced quite grandiosly that India was going to undertake a 10% of GDP stimulus and everyone was pleased because clearly India has been going through a very bad period, even prior to the pandemic. And the pandemic was wiping out about $5 billion a day of activity. It wasn't, we don't have a social safety net, etc. So people are excited about the 10% of GDP figure. And then in the next year or two when the details came out, it became clear that this was nothing of the thought that in fact they were counting all kinds of things like the central bank support, contingent liability support that had already been promised earlier, the support that was promised to farmers as well with part of this supported new package. And it came down to finally where the amounts that were actually being pledged, it was things like maybe somewhere between 0.7 to 1% of GDP. So there was a clear, there's just clear inadequacy and lack of capacity to think creatively on the part of this government. Having said that, there is the genuine question, do you have resources that you can garner? And there's two or three things that India does have which it's not really using. I wrote a piece on how India can finance some of its required fiscal stimulus. So obviously we are going to have to think of some level of monetization of the deficit. That has been something that has been taboo, but I think increasingly we're going to have to just face after that. There are particular sorts of simple things that we ought to be doing. Cutting interest rates now will mean that our debt to GDP ratios are much more stable in the medium term. And while that's happened, it's happened very inadequately at the moment. Then there are very simple things. We have what I call sessions here, you know, dedicated funds to particular outcomes. And they've just not been used, the education sets for example. There are many ways in which the government should they have been, should they have had the capacity and the bureaucratic capacity, to quite easily shift a certain percentage of GDP into this kind of essential activity that we need, but we have not been able to pull that off. And so it really is striking as to how the state governments are now being made to respond. And some of them have better capacity and are able to respond both in the resource front and really on the administrative front where others are not. So overall it is a mess, but it's a mess of governance as much as it is a mess of actually capacity. And Chao, how do you feel, which you might call, we're in a globalized world and things are organized as a nation state. How do you see whether it's related to issues of climate or multilateral engagement? The development of India in relation to the disease, in relation to the needs, and what you might call the retrenchment of multilateral concern. How does India fare in this context? So I think that there's a lot there and so perhaps we can start, maybe I can unpack that by starting with what I think are the contradictions of India's coronavirus pandemic response. When we think of countries in Europe especially with well-functioning, well-funded health systems with what I would describe only as an overall stability. I think that it's a fragile stability because I've been working recently on cases where many rich country governments like the UK's NHS for instance cannot afford some of the new drugs that are coming to market around cancers and diseases like cystic fibrosis. So it's a fragile and not quite certain stability. But there is evidence that there's a well-functioning system that by and large takes care of the public health needs of its population. We don't have that in India and I think that one of the most tragic contradictions of India's outsized response to the coronavirus pandemic is the question that Arjun and I tried to ask in our article for a major financial newspaper in the country, which is how is it that we have been able to sit on, completely overlook and accept what really is an unacceptable rate of completely preventable deaths. We're talking about 1500 children a day in infant mortality due to a lack of vaccinations. We're talking about 1200 people a day who died due to tuberculosis. We're talking about the millions. Literally this number is in the millions who die as a result of complications, usually respiratory connected to the disastrous apocalyptic air quality in many parts of this country. Arjun and I are lucky to live in a city, Bangalore, which actually does not have the same kind of industrial base or the farming base and therefore not the same kind of horrendous air quality that many parts of Northern India get. But we're nevertheless affected by it and we see it affecting our country long before the coronavirus pandemic. And so one of the contradictions has been a simple statistic, which is that more people die in one day of tuberculosis and a lack of vaccinations than it died in the last three months due to the coronavirus pandemic. And yet we haven't done anything close to that. We've spent up to or lost up to $5 billion a day in our lockdown due to decreased economic activity. We've spent an enormous amount of money coordinating very, very complex state and federal institutions to enforce and maintain this lockdown. And we've done all of that for a pandemic, which I think we are responding to primarily because it affects the rich. Now, this is not an argument for us not to respond to the pandemic. And I think we wanted to be clear about what we were trying to say as well. I think we support the government's response to the pandemic. We don't support the abject lack of planning that left millions of people destitute and dead as a result of it. But we support the effort that is being made in this pandemic. We just want to very clearly point out the glaring contradictions in this response. And as in terms of motivation, we are hard pressed to think of any reason other than that tuberculosis, bad air quality. These are things that primarily kill poor people. Of course, rich people are affected by air quality as well. But the way they're affected is that they have to cut short their evening walks in the park while wearing N95 masks. And go back to their homes that are powered by very, very expensive air conditioners and air purifiers of which there is a booming market in this country. Tuberculosis is an infectious disease. It very rarely affects rich people because rich people don't live around people who have TB. The way that TB has played out in the world over the last few decades is that it has been completely eradicated in many western countries as a result of which many parts of Europe and the United States, you don't even take the vaccines against TB that we take, the BCG vaccine. We continue to take the vaccine here. But the way it works in India is that between the rich and the poor, it's like the difference between Switzerland and a much poorer country in terms of how TB affects us. So it's a huge part of the life of the modern Indian state, life and death from TB. And it's no part of the life of rich people within this country. And as a result of which it's also almost no part of our fiscal response, our investment in curing and solving this disease. So that is a glaring contradiction. At the same time, I think that we, I think we can effectively use what's happening in the coronavirus pandemic to address those issues because they do form a part of the broader vulnerability that will make the pandemic worse in general. And it's part of a larger group of health factors that I think could possibly get some attention now, given that we are now paying attention to public health for whatever reason, even the fact that it's rich people who are involved and vulnerable in this situation. But I think, to me, is the most important way in which we can use the pandemic to pay attention to the broader issue that the pandemic addresses, which is public health in India and other poor countries with similar situations. Because we're not the only country where people are dying of diseases that can be cured, that we can invest in cures for at much higher rates than the coronavirus pandemic. And so I think that this applies not just to us, but to many, many parts of Asia and Africa and some parts of Latin America. I think that globally, I have a friend and a colleague called Astra Taylor who was talking about this in relation to the unmasking of the paradigm that you were talking about right at the outset of this conversation. And she had a typically wise and witty thing to say, which is there are so many rules we see being dismantled or ignored in order to combat and exit from the pandemic. And it does make you wonder as to what those rules were for when they were there. If we have to ignore them in order to exit the pandemic, why did we need them in the first place? And for me, I think the most important way in which that which is a manifest itself here is in terms of intellectual property. Later today, I'm going to be part of a WHO event to launch to formally launch a WHO COVID-19 technology pool for not just treatments and vaccines, but also diagnostics. Because again, since this is a new coronavirus, even things like testing kits and protective equipment come with intellectual property monopolies around them. Every pharmaceutical company CEO opposes this, but is having a hard time saying that publicly because suddenly now it's not poor people in India who are saying we need access to medicines. It's not poor people in a township outside Johannesburg who are asking for access to medicines. It's all Detonians who need access to medicines. You know, it's the British cabinet. It's the leadership of countries in like Spain and Canada and so on. There's a very, very rich, I mean, I just have to say rich white people are really affected by the coronavirus pandemic and care just as much about access to medicines and treatments as the poor people who are asking for it 20 years ago were in India and South Africa. And so pharmaceutical companies can't be as callous as they were. So all of them are saying, well, we're not going to profit off this pandemic.  There was a lot of attention paid to the increase in market capitalization of digital service firms, including Amazon and many of the media companies. I think less attention has been paid to the ways in which the only other industry that's genuinely come out of this better is the pharmaceutical industry, which has seen overall an enormous rise in market capitalization. Where you take the top, you know, six or eight companies. So despite the fact that they've already profited off the pandemic, they're all saying that they're not going to. At the same time, none of them have formally announced any renunciation of the very strict, strictly enforced intellectual property monopolies that they wield. They oppose the WHO COVID-19 technology pool, which would be a place where all monopoly ownership of anything that can get us out of this pandemic will be pulled in order that pretty much anyone can then make those things, whether it's a vaccine or a drug or a test kit. Now, it is really astonishing. And I think that somebody should talk to them publicly and more seriously about this to be able to put them to the test because either you're profiting from this or you're not. And if you're profiting from this, then you should face the consequences of it. And if you're not profiting from it, we should have a formal arrangement by which you can say so. And if we're not going to be enforcing intellectual property monopolies in a pandemic as a necessary precondition to exit the pandemic, then I would love to know why we have to enforce those monopolies in order to do everything else we do, which is survive cancer, which is survive hepatitis C, which is survive cystic fibrosis, survive tuberculosis or any other disease that we die off. Yeah. Yeah, it's almost as if what I, Arjun, you, you'll correct me if I get the economics wrong, but the inelasticity of demand of a person who is ill allows predators to exploit. And we just, we just can't have a system like this. There's something, how would I say, more important than the inspiration from profit at stake here. And I sense everybody feels it across the whole spectrum of health challenges and other things that are of essence. Let me, let me turn now. Many of the guests who have been on this podcast talk about climate as the thing that was haunting before the pandemic and fingers crossed after the vaccine and the other potential mitigating influences and remedies have been created on the horizon in a 10 to 15 year horizon, not much longer is a long, just serious gestation transformation of the very structure of energy so that man can can survive. And the reason I bring that to the table with the two of you is first, there's a quite a division among my guests between whether the climate initiative, which was gathering momentum prior to the pandemic will be spurred on by what we've called tonight the unmasking. Or, or will it be what you might call diminished in intensity, because we've used fiscal capacity at a very large level to fight the pandemic, because global cooperation is breaking down, particularly between the United States and China. And because people are exhausted, and the idea of a large transformation is daunting. And maybe they want to be quiet for a while before they embrace the next challenge. On the other side of that, what you might call a macro Keynesian might see a national and global program of energy transformation as a new source of aggregate demand and recovery and employment. And so I don't know how you see this, but one thing I do know is every climate expert or leader that I have interviewed feels that the fulcrum of what will matter is how the intersection between development of India, given the scale of the size of the country and transformation of energy in India. At the margin, they consider that the most important project. And obviously, given the level of development in India, it's a very, very, very hard thing to imagine them doing the public good for the world on their own without multilateral assistance. So with all of that context, how do you see the horizon vis-a-vis energy transformation addressing climate in the country of India? Arjun, why don't you start? Sure. So, Rob, you know, it's an interesting thing.  Last week, we had a cyclone that came in through and really kind of one of the most powerful cyclones we've had and flooded and kind of destroyed one of India's major cities, Kialkata. And now we have locusts which are coming in from Africa, right? So these are the so-called biblical plagues, you know, the locusts, flood plagues. And in some ways, maybe as someone said, you know, they're just symptoms of ecosystem collapse. We just keep repeating these as some sort of collective memory of last warning cries of, you know, some sort of collapsing civilization. Sometimes I think that, you know, all of these come together as warnings and this is what people have been worried about. So India, you know, in this context, in the context of climate change, obviously there are two competing camps, one which says that I think rightly and justly India should not be made to pay for the fact that the bank has been has been thieved and taken away by the West and maybe China to an extent. And there is, of course, the other camp, which is also right, which is that any sort of development plan cannot be innocent of climate change. And in India, we know that there is absolutely no way that development plans can ignore climate change. And therefore, I think there is a concerted effort among scientists and so on to think of different ways of doing this, which of course will mean technological transfers, different sort of financing and so on. I'd like to say a little bit, though, about the whole question about the financing of this transition and the worry about fiscal capacity. You know, it's true that we will have spent money on the pandemic, but I don't see the number itself, you know, let's say debt to GDP ratios go by 20%, 30%. I'm not sure exactly what the number is. What is the main concern that people have? I think the main concern that people have is that it will in some ways use up the real resources that we have and create inflation. A friend once said this, you know, better the economy run hot than the planet. And I feel like that is true that we have to try to make sure that all our resources are maximized. At the current moment, I don't see there to be a problem post the pandemic of resource unavailability, real resource unavailability. There are going to be people who are unemployed, there are going to be people and systems which have to be rethought. And so I actually see it as an opportunity globally, and of course as well in India, to rethink our ways. So you said your guests had two kind of competing views on sort of firmly in the view that this is actually an opportunity to think creatively and effectively about the climate, which must happen if we're going to provide as a species. Charles, how do you, how do you see this challenge? I would say the similar to similarly to Arjun, but with some conditions attached. And I think let me just explain what I mean. I've been a public health activist and I think that there are some ways in which in a country like India, even despite the global conversation around climate change, it can seem like a both distant thing. I think that is far away in the future. It is for me, I think that there is a way in which we have dire immediate effects of energy consumption that are costing human lives. And I think that pollution is one of the most vivid examples of how that's affecting us. And I think the condition that I was going to talk about was to be able to cast efforts at future climate change prevention and containment around to cast that conversation as a question of saving human life. But not in the future, but in the present. You know, we have one of the consequences of countries like China and India deciding that they must do their own bit around carbon depletion to catch up with three or four centuries of work in the richer countries in the world has been the fact that it is certainly while helping their economies grow has been costing enormous human lives within these countries as well. And I think that we don't pay enough attention to that fact. We don't publicize or examine the number of millions of people in India that are being mortally affected by our energy consumption. And I think that a very, very good way to start without necessarily even using the term climate change would be a public health authority that is designated to save human lives in India that come at the cost of air pollution and bad air quality in the country. The regulations that we can design around that the rules that we could put in place to combat that, you know, whether it's incentivizing electric vehicles in whether it's it's creating massive new investments and infrastructure for public transport would be would go a long way in terms of our larger longer fight against climate change. But I do think that casting these these these plans I think for it to be politically feasible either for the government or even expedient for any government this one or even a more left leading one to undertake. I think to use the cost in immediate human lives in the present day as the best justification to start implementing these programs is is a really good one. It would make sense at every level possible. And it's one that I'd support enormously. And I think would make sense to, you know, every voter in this country. And I'm surprised that that that we haven't started implementing programs that use human life as a basis to address climate change. Yeah. Well, what, what do you see in the world? You look at this tension between the United States and China. And our capacity for having a collaborative global system is how we say it appears to me to be at risk. But I know that in India's relationship with China, there are some there are some other complexities. And I'm curious how you see the prospects for India in light of the breakdown of the relationship between the United States and China. Do either of you see, how would I say a dreadful scenario that relates to that? Or is that not such a concern for people in India? I could start with perhaps at the very moment, India and China engaged in a territorial standoff. So in the middle of the coronavirus pandemic, Indian soldiers and Chinese soldiers beating each other up and shooting at each other in one remote corner of India over contested territory. There's a long standing territorial dispute, which is, it's just too slow burning to even care about. But they're in the middle of doing that right now. Indian media is awash with what I can only think of, what I can only term as completely delusional articles, which suggests somehow that the Indian government is going to turn India into a manufacturing base that will take advantage of the United States and the world's anger against China. And it's perceived inability to manufacture now because of the coronavirus pandemic, which is complete nonsense and has been described as such, I think by anyone with a working mind. I think that the larger question of multilateral cooperation that you brought up, I think is very important. The piece that Arjun and Joe and I wrote together in Project Syndicate talks about the flu shot that we all take. So this is the shot that Americans take every year, that not as many Indians take every year, but is available to us to take every year. The newest version of that vaccine, it's called a quadrivalent vaccine. So it targets, I think, four different strains of the influenza virus. That shot that is manufactured by hundreds of companies around the world because there's no monopoly on it is produced through collaborative research that takes place every day through national monitoring laboratories in 110 countries around the world. And it's based out of a center, a very little known center at the WHO. It's called GISRS, the Global Influenza Surveillance and Response System in the WHO. So what they do is they monitor all these flu strains and then twice a year at the opening of the flu season in the northern hemisphere. And in the southern hemisphere, they issue recommendations as to what strains should be included in that year's flu shot. Obviously, the flu shot, as we know, is not something that eliminates your susceptibility to influenza. It merely reduces it by a substantial extent to the extent that it is recommended by health systems almost everywhere in the world. So what the WHO does is that it uses international cooperation, a vast network of international cooperation to analyze on an ongoing basis of flu data and uses that to create an open source formula that then anyone can apply and produce. Now, no country government is bound to follow the WHO's recommendations, but invariably every one of them does, including the Center for Disease Control in the United States. This is the model of multilateral cooperation that we have had. And incidentally, one other curious factor about the flu vaccine is that, of course, the very first strains of the flu vaccine were developed by the very same Jonah Sock, who gave us the polio vaccine and who, of course, memorably said to Ed Marrow in an interview when asked why he didn't patent it and have a monopoly on the polio vaccine, said, I never thought of it, it was people's vaccine, can you patent the sun? Which really brings a tear to my eye even after all these years. So the model of multilateral cooperation, it's not even a fantasy that we have to imagine and I think that's the point we were trying to make in that piece. This is how one important part of the world already works. This is how it's worked since 1973, in fact, right? And it's done a lot of good in the world. It works. There's a system in place to do it. We could replicate that and build on that. Use that as a model for not just flu vaccines, but for pretty much every other form of disease research and development that the world needs. We don't actually need to follow the model where everything, every aspect of research is then divested with no responsibility to a private corporation to bring to market. So we have a system in the world that works. It's an important multilateral system. It has to work. And to the extent that the coronavirus pandemic increases multilateral cooperation, despite the United States recalcitrant to do so, I think that could be one of the best outcomes of this. Not only the fact that each of us is individually interconnected in terms of our health affecting the other, but the fact that the more that countries cooperate in this interdependence and in this in this mutual feeling of solidarity. The better results we will come to, whether it's containing a pandemic or curing it. Arjun, any thought? You know the question about where India will be with respect to these global changes that are going on. Yes, it is interesting, India's relationship with China has always been fraught since the 60s at least and it's seen itself as a rival though it's never really I think measured up to that in the economics here. It's hard to say, Rob, I, you know, this is the world in which I think we're suddenly seeing a move to greater nationalism and a certain belief that the nation state has to be strengthened and that you see across the world. So you have these contradictory impulses we do understand that the world requires global cooperation. Many of the responses have been very national and nationalistic. India's, the new word that has been bended about is self-reliance in India, which is of course a harkening back to 30, 40, 50 years ago. So it's hard to say where India will end up in all of this, whether at the end of 16, 18 months we'll actually, you know, try to be much more reliant on our own supply chains or whether, you know, we'll return to the earlier world with respect to manufacturing. So I wish I had more clarity on this, but I do think that's an axis on which a lot will depend on the way we go forward. And my hope as, you know, I have in that article with Achal and Joe is that we actually think of the good parts about global integration and try to, you know, promote those as opposed to going back to a nationalism which is narrow, inefficient and probably in the end self-destructive. Okay. Well, I want to thank you both. We've explored a lot of terrain in the issue space, your deep dives in the realm of health care and the systems that we create, create incentives or disincentives to the protection of health. And the context of India are all very important dimensions for those affiliated and interested in the Institute for New Economic Thinking to understand and to explore. I hope that both of you will come back and see me again in a few months time. Because I think this is a work in progress. I think you're both quite a bit ahead of the curve. But there'll be new stimulants. There'll be new turns in the road. And I guess for right now, I want to thank you for steering our minds tonight. But I do hope to see you again soon. Thank you, Rob. Thank you very much, Rob. And thanks for everything. Thank you. We'll be in touch and we'll be back together before too much time passes. It was a pleasure. And check out more from the Institute for New Economic Thinking at InetEconomics.org.


everyone shiver 1000 here today we've got a parcel in the mail and as you can tell by the title it's a set of mics well one mic so let's open this up and check it out okay guys so like I said this is a mic it's for my phone it's a wireless mic it's a lavalier mic but there's no wires to it so I was reached out to by company wanted me to test this I get to keep it and I will what is it it's the Deco's audio store and they're out of Temple Terrace Florida D-I-K-O-S audio store so and here's what's in the box this wireless lavalier microphone it's the X0039BL58J it's made in China wireless lavalier microphone for Android they also have these for Apple too guys so let's get this open up here now we're gonna try it out I I have seen seen a couple people doing these on YouTube not not this exact brand but similar I don't know if this is a knockoff or what but this looks identical to the ones that they were using or reviewing plug and play 20 meters of accessible reception precise radio clear TIMBRE timber I don't know what that is high sensitivity and widely compatible like I said this is for the Android got a nice case this is the cable for it get it out first like said they do make a frapple from what I seen on the reviews they seem to work pretty good there's the charging cable there's the instructions down in the bottom and here's the unit it's inside here it charges and once you put it in kind of like the the earbuds I did those earbuds and I ended up selling them for like 12 bucks there's an introduction tells you everything about it there connect the receiver to phone with the port I'll show you all this here in just a minute guys press the switch key to microphone the indicator light turns green and then power is turned on this is all in this is all in English the indicator the indicator red light of receiver keeps always on should say stays always stays on or the open then open the phone the phone camera live app or other apps and then use the microphone to receive sound special note for the use of port there is no battery in receiver and no need to recharge during live if the phone is low-powered during live it can be recharged directly through the type C port okay so let's open this up see what we get in here do that out of the way let me bring in a little bit farther a little bit closer there and it's opening these up and there we have it so this does not need to be charged just type C it will go in to the bottom of my phone right here and here's the mic it's got a magnetic lid on it so that's pretty nice I'll pull this out does have a clip on here like I said a lot of people done not a lot but some people's most most of people that's reviewed these their complaint was that they couldn't swivel this which I agree this should be able to be swiveled but it's not a dear deal breaker for me because usually I like to have my mics right up straight up down like towards my throat or towards my mouth so I mean this is just fine with me but you know teach his own let's go ahead and pull this off of here because it's our charging contacts so must be able to charge it there too so if I need to recharge my phone during a live stream I don't have to pull this out okay once this is stuck in there because there's where you plug it in at so that's pretty cool there it's in there like that so if I need to recharge I don't have to unplug it just plug my USB cable right into there my USB C cable right into there the USB C cable here you can charge it like that I guess so but yeah it's supposed to charge as soon as you drop it down in here of course gotta be plugged in but yeah so I don't know if it's gonna yeah it does have charge in it okay well so this plugs directly in here I don't know if there's anything else I got to do so let me check up and see I don't think I've got a pair of this or anything I think this just automatically talks to the receiver very nice but let me make sure and let me make sure I'm getting sound into here and then we'll try it out okay guys so this is the phone without the camera phone without the mic this is just the built-in mic man this thing looks awful shaky it's got that digital video stabilization but doesn't seem to work right on these phones but anyway this is before the mic now let's go ahead and plug the mic in okay guys I'm gonna put this on now I didn't measure but I pasted out to 20 yards about 20 meters which is about 60 feet so I'm gonna walk to the end of the road and back I myself I don't know about anyone else but I don't know I mean I'll never be using it that far away 60 feet is a long ways away so you know to be filming I really can't see myself going that far or anyone else for that matter but let's see okay I was just checking my phone holder so I won't know until I upload this video or until I check the video out on my editor so as you can see I'm waving my arms I'm still talking we're going out here to the street okay so this is approximately 60 feet right here so let's go a little further one two three four five six seven eight nine ten and I don't know if you can hear me from here so that's 30 more feet so I'm 90 feet away but the claims does not it does not claim it'll go 90 feet I just wanted to see if it would so they do make wind the little wind socks the little fuzzy things that go on the mic so sorry for wind but anyway there you go so I'm gonna go in and check this out on check the footage out on a computer let's see how well it did and then I'll come back with you with my final thoughts stick around guys okay guys so there you go that's the mic like you said I don't know what the actual name name of this is just as wireless microphone same with on the back so I'm not sure I mean I'll put a link in the description for you in case you're interested these are on Amazon but I do have a link for it where does thing go all it says is the wireless lavalier microphone for Android like I said again they have for this is from this is made in China X0039BL58J so I will put a link in the description for you I did find an issue with this the issue I found was when I went to plug my phone plug plug my charger into the into the receiver my phone brought up it wasn't the it wasn't the right charger for this phone wouldn't charge it I don't know if you guys will have that problem or not that needs to be straightened out because you know it's what a lot of people mainly use these microphones for is live streaming and if they can't charge a phone while they're live streaming well you know that's kind of that is kind of a deal breaker and it would be for me too fortunately I don't do a lot of live streaming I am on the the my phone right now using the mic right here see I'm using this mic make sure I had a right side up here's my other camera so yeah I kind of I don't know if maybe you can switch chargers and try it and see I will do that but as of right now just like say the typical everyday person is not want to know to check other things like that so right out of the box it doesn't want to charge my phone I don't like that everything else I do like I do like about it I will put a link in their description it's cheap Mike guys I think it would have ended up being right around 30 bucks for shipping and handling and excuse me and tax so it's like 20 some dollars I mean do I recommend it yeah I do if you keep that in mind that it may not charge your phone while you're using it keep that in mind because you know like I said I it won't charge mine and I'm not going to mess around with it and tell you you got to do this and that and the other right now in this video because you shouldn't really have to if it says it will it should but it didn't so other than that yeah I like them I like the case that came in I wish they would make one I don't know if they do or not I'll have to ask them I didn't ask them but if they make one for for this camera I I will get one immediately because I do like it it's very little very very light I mean this is you know it's very very light I don't even know it's on there and I think it sounds good and it even went you know 30 feet more than what it should have than what the claims are so that's that's pretty good now don't go out there and say well Shea Bear said that it will do that mine won't well it's not supposed to they don't claim it will I just wanted to try it to see it and it did so you know it will go a little further than then and and as you heard it's it didn't break up or anything so I mean it sounded pretty good all the way up to 90 feet which I was kind of high far stepping it so it might have been closer to a hundred but 60 feet I mean that's a good ways of way to be filming anyway like I said don't don't say well just because Shea Bear said that's not the way it works I just wanted to see you know for myself if it would go above that or if you actually got to be real careful I mean so you can kind of gauge it of where you're going to be filming out and I think everything will be alright for you so anyway guys I appreciate you watching and I appreciate them sending this to me but like I said there that issue does need to be addressed but I will give it a two thumbs up I will give it a four out of five stars it would have gotten a five out of five had it charged my phone like it should and like I said I'm gonna see if they make these for a regular video camera if they do I will have one because what I like about it is it's completely wireless I don't have a wire running down from here down to my belt or you know my pocket or whatever so anyway guys I appreciate you watching and again Shea Bear the myth man of the edge I'm gone for now we'll see you guys in the next one thanks again and bye bye


Thank you everyone for coming. So before I get started I want to just ask a quick question which is how many people in the audience right now either already use big data or want to learn how to use big data so they can put it on their resume so they can get three times more recruiter emails? Exactly, perfect. All right, that's pretty much what I'm going to talk about today, how to optimize your recruiter emails. Building data driven products using Ruby. So you're probably wondering who is this guy, why should I listen to him, right? He looks like he's about 12 years old, his voice hasn't even broken yet. So I studied computer science and bioinformatics at UCSD before I eventually dropped out so I could kind of join the startup scene in San Francisco and right now I'm currently a data scientist at ShareThrough which basically means I'm an engineer who also happens to be good at math and since I live in Silicon Valley they decided to call me a data scientist. The company I work for ShareThrough is a native video advertising platform. What that means is we consume large amounts of data about users all over the web so that we can customize ad experiences to hopefully make ads as a whole suck less, right? Which is good because it means I basically get paid to use data to improve my business's bottom line so really you want to listen to me because somebody pays me to do this so I must kind of know what I'm talking about at least. So my goal of this talk is to help you answer the following four questions. What is a data driven product? What does the development cycle look like for a data driven product? Where does Ruby fit in this new world of data science? And how can Ruby be improved to stay relevant in the age of big data? But I just want to let you know I'm not gonna talk about whether you should use like support vector machines or some new type of regression or whether you want to use principal component analysis. This is really about talking about how Ruby fits in and how we build data driven products as a whole. So before I get started I want to give you a couple of warnings. Ruby is not your only option, right? The world of big data right now is pretty much a minefield. You have so many things to choose from, right? We have Hive, Pig, there's R, Scala, Cascading, Python, Java, right? All of these tools in this ecosystem right now and really you're not gonna use just one of them. So my day-to-day job I actually use a combination of Ruby, Python, Java, R and even a little bit of Scala just to do what I do on a day-to-day basis, right? The key is it's all about picking the right tool for the right job. But since this is a Ruby conference I'm really gonna talk about the ways I see Ruby fitting in and the places in the data-driven product cycle where Ruby really is a good fit. So now we've got those warnings out of the way. We'll first start with what is a data-driven product? Right? A data-driven product is really anything that uses data to improve the bottom line of your business, right? So it could be a standalone product, right? Where the whole company just does data. Some examples of these are like Boundary, Mixpanel, right? Google to a certain extent. But I think it's more interesting to look at the ways you can incorporate like particular data-driven products into a larger company offering, right? Examples might be like ad targeting, right? Product recommendations if you're amazon.com, right? Which books you bought? Information, aggregation and filtering. So if you go on a news website which articles you're likely to wanna watch, right? We can see some examples of these all over the web at the moment, right? We have GitHub, the classic example, right? This kind of seems simple and trivial. When you really think about it, what GitHub is doing is they're giving you data and they're letting you use data to understand how you interact with their product and in understanding how you interact with your product they know you're gonna become like more attached to the platform as a whole, right? So using this simple graph, I can do things like know when my engineering team is most productive, right? And if I know when my engineering team is most productive, I can make sure I don't schedule meetings during those times. At the same time, I can check to make sure I'm not burning my engineers out, making sure they're not committing at one o'clock in the morning every single Saturday because nobody really wants that, right? The other classic example is LinkedIn's people you may know, right? So social networks realized a long time ago that a user's interaction with their network was highly coupled to how dense their social network was on that given network. So in order to help improve that, what LinkedIn did is they kind of pioneered this notion of people you may know, right? So we use algorithms and we use data based on how you interact with the network. In order to target people they think you're likely to want to get to know. In doing this, they're able to help improve engagement and overall improve the value of their product, right? Because LinkedIn makes money off advertising.  And the classic example is advertising, right? Like Google AdWords is probably one of the ultimate data-driven products, right? They use data from your searching history. They use the actual search term itself all to target advertising to you, right? And the cool thing is by using this data they're able to create value for Google as a whole which lets them deliver the awesome search results that we're so used to on a daily basis, right? Using this data product they're able to make money for Google. But it doesn't have to be a product in the traditional sense. Another great thing you can do with data is improve marketing for your company as a whole, right? This is a great example. Facebook released this graphic back in December of 2010 and they got tons of pickup, right? All the tech blogs covered it was on TechCrunch Mashable and like a bunch of people use it as their screensaver, right? Just a simple product like this you wouldn't normally think of it as a product but when you kind of take that step back what it's doing is it's giving Facebook a way to reach users, right? And if users see how connected Facebook as as a whole they're more likely to want to join the network and most importantly brands now realize how connected Facebook is because a brand manager sees this.  So that's great. You've told me exactly what a data-driven product is filled with lots of buzzwords but I'm an engineer so I wanna know how do I actually go about building something? So I think building a data-driven product kind of comes down to this cycle which has four major steps, right? You start off with asking the right question. There's collecting and cleaning your data then you move on to building the predictive model and finally you get to like publishing your results phase, right? The important thing to notice about this is this is not a linear flow, right? It's not waterfall, building data-driven products is just like building every other product that we're so used to building, right? Like Rails, web apps, we all do the agile thing, right? It's very much the same with data but it's very hard to get out of that traditional like waterfall straight down model, right? Like most people in a traditional kind of enterprise background doing data stuff using traditional like business intelligence tools, teradata, that kind of thing. You get in this kind of waterfall mode, right? You just like give me a question, I'm gonna answer it, same technology all the time. But I think it's important to realize that to build good data products you need this cycle, right? Cause you might finally get to the publish results phase like you print out this graph and then the business says, okay that's cool but we've kind of pivoted since then so your graph really has no value now so you need to go back and do it again, right? Same thing happens like you've built the ultimate model, tons of work and then somebody in business development signs a new deal with a third party and all of a sudden you need to integrate that new third party's data source cause hey the business needs it, we need to put it on our website, we need you to put it in your model. All of a sudden you're gonna have to go from that whole building a model phase back to cleaning and collecting data because until you've done that you can't truly build a model. So now that we have kind of the four main phases outlined we're gonna like take a step in and look at how we do each one. So the first phase is all about asking the right question, right? This seems really simple, it seems kind of trivial. You're probably wondering why I'm even talking about this, right? This is a tech conference I said I was gonna talk about Ruby. This is actually one of the hardest phases, right? If you don't ask the right question no matter how awesome your tech stack is no matter what technology you pick it won't matter, right? If you're not answering a question that really helps the business you're not delivering value and the whole point of like engineering and product is delivering value to your business, right? Which means you really need to focus on asking the right question. And conveniently the only thing you need to do that is English, right? You don't need Ruby, you don't need Python, you don't need Java, you don't need Hadoop to help ask the right question. What you need to do is you need to go out and you need to talk to the business, right? You need business context. You need to know like what makes a business run. Like what are partnerships looking like? What does the market look like as a whole, right? All these things are traditionally the hardest things for engineers to do, right? We actually have to go out and talk to people which is quite challenging. I don't personally like doing that that much. We have to do them so that we can kind of get that first phase done. So we can use Ruby to kind of help with that do some exploratory stats but really at the end of the day, that whole first phase, you don't need any programming technology. You just need to talk to people. So for a personal example, right? I kind of want to guide this whole process through what I do on a day-to-day basis. So the first example that I really have is the marketing department in the business comes to me and they say, okay, Ryan, I want a data dump of percent of users on publisher X that I've also seen on publisher Y, right? So I want to see how many users I've seen on Forbes that I later saw on the all or business insider, right? I could do that. That's simple. That's like a very easy question but the problem is what value does that really give them, right? A data dump is very simple. So you kind of, you take that step back which is the next box down which is the thing they're really trying to ask, the question they really want to know is what is the value of a user on an ad network, right? Because if we can determine the value of a user on an ad network, we can better predict our revenue as a whole and we can better gauge how much we charge an advertiser for each impression, right? But that's almost too big of a question, right? Like now I found this huge theme. What do I actually do with it? I can't really answer that. So you maybe take that one step down, right? Which is in this case, what is the supply of a user of a given type, right? So given a user's been seen on publisher A, what's the supply of users who will also be seen on publisher B, right? And most importantly, can we predict given we've seen a user on one publisher that we'll see a user on another publisher? And that's kind of the question we chose to work on. So once we have that question formatted, we get down to the real code writing phase, which is phase two, which is data collection and cleaning. So this is not very glamorous, but you'll spend 90% of your time doing data collection and cleaning, right? No matter what anyone tells you, right? Like in computer science class, they're always like, focus on math, focus on stats, that's all great. You're gonna spend 90% of your time cleaning data. Data in the real world is very, very messy. And the thing that really makes a difference between a good data product, a good engineer, a good data scientist, and someone who isn't very good is your ability to deal with and clean data. So for example, you would start off with something like this, right? This is what my logs look like, that I do most of my analysis on, right? You can kind of read them, but you can already see, like there's some missing values. We have the intentionally blank HTTP referer. That's awesome. And the whole thing is just kind of messy, right? And what I need to do is I need to take this massive data and I need to output something that looks like this, because I need this two-column CSV so I can input it into my graph algorithms to actually determine what percentage of a user's gonna cross over, to build the product that the business really needs, which is a predictive model for how many users are gonna be seen across our network, right? But for you guys, how do you get your data, right? Where does data come from? Right now, in the social web, most data comes from these four sources, right? We have server logs, right? Your front-end boxes, Rails boxes, engine X boxes, they're all producing tons and tons of logs. You have that, you have third-party APIs, right? Everyone loves to collect Twitter data right now. Everyone loves to collect Facebook data. You have web scraping, right?   And finally, we have direct user input, right? You have a questionnaire, you have a survey, something where the user's directly giving you data. The important thing to note about these four sources that's very different from what we're used to is they all require programming skills, right? Like none of this data is just conveniently handed to you, right? Like everything requires you to go out and write code to get it, right? And it could be even worse. What happens if someone gives you a PDF, right? They're like, hey, I gave you data. You're like, no, you gave me a PDF. I can't do anything with that. But ultimately, you're gonna have to pull data from all these sources in order to even start building a real product that helps your business. And this is really where Ruby comes in, right? Ruby has tons of tools to make it possible for you to collect and clean data, right? This is just a sample of the tools I use on a day-to-day basis, right? We have NoCoGiri. If you wanna parse XML, parse HTML, right? If you're doing any sort of scraping, you're gonna spend tons of time using NoCoGiri. And even if you're dealing with old APIs that still use XML, you're gonna spend tons of time using NoCoGiri. We have SaveOn, which everyone loves, right? It's a SOAP client. If you're having to use an API that was built in the 90s and they haven't updated it to REST, you have to use SOAP. SaveOn will make it a lot easier, right? We have REST Client, which makes it really easy to make HTTP requests, because you're gonna be making a lot of HTTP requests if you wanna cross reference data sources, right? Pretty much every API right now is HTTP-based, especially if it's modern and using REST, so that'll come in very handy, right? We have PDF Reader, because you will most likely have somebody come to you and want you to do some analysis on a PDF. Like, hey, I have tons of data on a PDF. I need you to extract it, right? Ruby makes that possible. And then finally, we have Sinatra, right? Like, Sinatra's a great way if you just wanna quickly set up a survey, right? Like, I could write a survey that I can put up on an Amazon Mechanical Cirque to ask somebody their opinion on the election in probably 10 minutes, right? And then we also have Twitter. Twitter is a classic example, and that's where we're gonna look at our first piece of code. So maybe a simple thing you wanna do is you wanna say what's the word frequency from Hurricane Sandy, right? And the awesome thing about Ruby is we can pretty much do that in, like, 11 lines of code given that we've already, like, configured our tweet stream client, right? And if you just take a second to read that code, what you can see is that it's really very basic, right? Ruby takes this hard process of cleaning and collecting data and makes it much easier to work with, right? So here we're already able to do, like, basically the two hardest things, which is one, go out and collect the data, right? We're able to track all the keywords that have the hashtag Sandy in them, but it just doesn't just do that. We're also getting the cleaning phase done, right? Because no algorithm that's gonna do interesting things on natural language is gonna take, like, really sentences, right? Most of the models fundamentally want words or engrams or some sort of representation, and we can already get that, right? Ruby makes it so easy. We're able to take that entire sentence, split it, and then remove blank strings, like, just like that, right? Which means now we can simply output a comma-separated list of words, which would be awesome to input into the next phase in our algorithm. And when we talk about collecting data, we can't not talk about Rails, right? Rails is so easy if you need to collect direct user input, right? Like, if you wanna use Mechanical Turk, right, for sentiment analysis, anything like that, Rails makes that so easy. There's an awesome open-source project from Twitter called Clockwork Raven, right? Which has gotten a lot of press. It's basically a way to submit jobs to Amazon Mechanical Turk, get feedback, and then refine which users you think do the best job, right? So that's all written in Rails, and there was a funny quote the other day from one of their data scientists. He said, I was trained as a classical scientist, but I spend most of my time writing Rails. Because what it comes down to is he spends most of his time writing the frameworks he needs to collect the data so that he can even do the complex analysis that he spent all that time in school learning how to do. Like, that's great, but he said you're gonna talk about big data, and my data is big data, right? Everyone right now wants to say their data's bigger than everyone else's, and it's kind of, yeah, we have to get used to this buzzword and realizing that just because your data's big data doesn't mean you can't use Ruby, right? Everyone says, well, Ruby can't scale. It turns out some smart guys created this thing called Hadoop, which means you can make Ruby scale, right? So Hadoop is Java. I don't really like writing Java personally, so if I can get away with it, I'm gonna not write Java. And for a lot of tasks, you don't really need Java in order to write Hadoop jobs. So if you remember those log lines from before, when there's two of them, it's pretty easy to see which pieces of data are missing, right? We can see that it's pretty easy, blank HTTP refer, there's a couple of missing values. But in reality, in production, if you're dealing with big data, your logs basically look like that, right? It's an undiscernible mass of text. There's no way you can go through and manually inspect every single log line to see like, hey, somebody introduced a bug in the client, and all of a sudden, my data's not being collected, right? And that's kind of where Hadoop comes in, and that's where we can really use the power of Ruby, because Ruby makes it very easy to do simple tasks across humongous clusters of nodes. So here's an example, from those log lines, I happen to know that at a given point, there was a bug introduced in our client, and lines were being passed back to me without user IDs. And without user IDs, there's not really much interesting stuff you can do if you're an ad company, because you need to track users. But in basically three lines of real Ruby plus a header, we can basically take 10 billion log lines and just output the log lines that have missing user IDs, right? And if that might only be 10,000 log lines, right? So what I can do is I can use Ruby to take this massive data that's basically indiscernible, and distill it down into something small, right? Once I have those 10,000 log lines, I can probably download those onto my local machine, see like what time did the log lines start appearing, right? What time was the bug introduced? What time was the bug solved, right? One of the best things you can try and do, one of the things Ruby's so good at is taking big data and making it small, right? Because you really want to make your data much smaller to make it much easier to work with. So when it comes to Ruby and Hadoop, there's kind of three good options at this point. There might be more, but these are the kind of three mature options that we've experimented with and used, right? First off, you have vanilla Hadoop streaming. That's a script I just showed you. Basically, Hadoop will take every line that it sees as input and just output it to standard out. And then your Ruby script can just read it in and do whatever you want with it, right? This gives you ultimate flexibility and power, but tons of boilerplate code to write. So if your data is serialized, you have to deal with all the deserialization, right? Trivial things like I want to do group buy an account, right? That becomes hard because all of a sudden you have to write your own Ruby code to do a group buy an account. And it's a distributed account, so it's very hard. Conveniently, the guys at Infochimps wrote this cool library called Wukong, which is an abstraction on top of the vanilla Hadoop streaming, which makes it much easier to work with Ruby on Hadoop if you just want to use the streaming and regular Ruby. And what they do is they give you this kind of abstraction that makes it very easy to perform classic tuple operations, right? Something like a group buy account, a distinct, all those kind of things become very easy. But streaming is not necessarily the most efficient way to use big data. It's definitely not the fastest way you can run a Hadoop job. Really what you want is you want to drop down to Java, right? You want to drop down to the native libraries. And those cool Java libraries pretty heavily adopted. Everyone really uses it these days called cascading. Cascading is all Java and there's tons of verbosity. It makes it very easy to do the same kind of tuple operations, right? Group buy, account, those kind of things. But the problem is, is Java. So I end up writing like 100 lines of constructors, like abstract factory-factory group buy, like awesome. So the guys at Etsy wrote this cool wrapper, cascading JRuby, right? And what it lets you do is it lets you write JRuby scripts that end up translating down essentially to JVM byte code, right? So really the power of Hadoop, the power of big data is JRuby, right? JRuby is an awesome project. It gives you the full power of the JVM and it pretty much lets you do anything with Hadoop. You want to do in Ruby syntax, right? Sure you'll take a performance penalty, but most of us aren't running on like Facebook and Google scale, right? We don't have like petabytes and Zeta bytes. We have like maybe terabytes, right? JRuby's fine for that. You can use JRuby to write pig UDFs, right? Maybe everyone already likes pig. Maybe your company already has a Hive cluster. You can use JRuby to write Hive UDFs, right? Like custom Hive functions. Say I want to do something like geocode and IP address, right? That becomes a lot easier when you can write that using existing Ruby libraries. And much like the way Square rolls everything up before deploying it using their framework, you can kind of do the same thing with Hadoop, right? You just bundle up your JRuby in a jar.  And Hadoop is fine to run your JRuby. And we can really see the power of that with this example. So I'll let you read it and see if you can get the gist for what's going on. So yeah, this script basically is the classic word count example, right? So to give you a little bit of context, the classic word count example, if you use the raw Hadoop API, I think ends up being a couple hundred lines of code. If you use just vanilla cascading and you really compress it, it'll probably look a little bit shorter than this, but it's a lot harder to understand, right? The cool thing here is this is all the Ruby code we're so used to doing, right? Ruby gives us blocks. We can pass them around.  I have to create a class just for a custom function. This script basically lets us take however much data we want, right? Split it into words and count them, right? Something we all know how to do on the command line. JRuby gives you the ability to harness that across a cluster of a billion nodes if you haven't. So Ruby as a whole is a powerful tool for data collection and cleaning, right? We all know how to use data to, we all know how to use Ruby to clean data in Unix, right? Really, the great thing about it is you can use those exact same tools you're used to using every single day to build like data-driven products, right? Just because you have big data doesn't mean you can't do the stuff you're so used to doing, right? You write a script, you can run it on Hadoop. You want to use JRuby, you can harness the full power of Hadoop, right? Which just makes Ruby so powerful. When you combine that with the ability to easily collect data using things like Rails, it's such a natural fit for this hardest part of building a data-driven product because you will spend 90% of your time doing it. So if you can use Ruby for the thing that you spend 90% of your time doing, you already have a win, really, right? So once it's all cleaned, you kind of move on to the next phase, right? That's the statistical modeling and the prediction. That's kind of the glamorous phase, right? If you go and interview for a job, this is what they're gonna ask you all the questions about, right? Like what kind of distribution does my data have? All that kind of stuff. So it's definitely the glamorous part and it turns out really what you're trying to say in my personal example, right? I said I need to be able to predict a user's likelihood of being on multiple publishers, right? And what that really means is I need a function that takes as input a user ID and a publisher X and I need it to output me the probability that the user will be on publisher Y, right? So it's pretty basic, right? That's a pretty simple function, but it's gonna require me to do some stats, right? Like that's where the kind of data stuff really comes in, right? I'm gonna have to do some stats.   But it turns out, who cares, right? Like most of your time is gonna be spent cleaning and collecting your data anyway. So the fact that Ruby might not be the best statistical programming language doesn't really matter. The other thing you can always say is have you actually tried running R in production? It sucks, right? Like to take code and actually build a real product that your business is gonna make money on, you need to be able to monitor it. You need to be able to deploy it easily. You need to know when an exception occurs. All those things that Ruby already gives you, you don't get when you try and run languages such as R. You can do it, but it's definitely not the best way. And since we're all Rubyists, we wanna know how we can use Ruby to kind of do the statistical modeling. So it turns out what people may say, Ruby sucks at stats. There's actually a pretty good selection of libraries that let you do most of the stats you only need to do to build the kind of products we're talking about, right? Stat sample is an awesome library. It basically goes through and implements most of the statistics functions you'd ever really need, right? Sure, it will maybe lack some obscure genetics algorithm, but at that point, you should probably be using another language anyway. We also have a PsyRuby, right? Which is a big push, kind of an attempt to make Ruby equivalent to PsyPy. And what it lets you do is that same stuff, right? It provides you with matrix libraries, right? All these things that we need to do stats are there in Ruby, right? Like we can use them. And then most importantly, we have libSVM, right? So support vector machines are one of the most popular ways to do classification right now. And there's a pretty battle-hardened production ready C implementation of SVM called libSVM. And there's a Ruby binding for it, right? So if you just go online and you Google Ruby libSVM, you're gonna find a good blog post by, I think, Ilya Grigorik on how to use it. And you'll just find tons of documentation, right? There's no reason we can't use these libraries because most of them are in C. There's Ruby wrappers on top of most of these hardcore libraries like libSVM. And if you really need to do some obscure stats, we have libraries like RSRuby and RINRuby, right? That basically lets you send Ruby objects over to R. So if you need to do some obscure complicated regression, right? Like some least-angles regression that isn't Ruby yet, but you already have your data in Ruby, like from ActiveRecord, say, right? You run a BigQuery, you have all these objects. You can just pass those over to R. R can do the hard part. You can just put a little bit of statistics in, crunching you need, and then give it back to you, right? And again, the other tool we really have is JRuby, right? I can't say enough, like JRuby really gives you the power to build data products in Ruby, right? You can harness the full power of the JVM. And when it comes to data stuff, there's a JVM library for pretty much everything, right? Like there's already a Java library for hard matrix stuff, sparse matrices, pretty much every type of machine learning algorithm you wanna use, every kernel density method you wanna use, they're already implemented in Java. And the great thing about JRuby is it lets us write this nice wrapper on the top of that stuff that we already, in a language you already all know, right? We already know how to use Ruby. And since the hard part has usually already been written, there's no reason we can't put Ruby on top of it, right? There's no reason we can't make it so that data products are approachable to all of our engineers. So really, although everyone will make fun of you for it, there's no reason you can't do statistical modeling in Ruby. And once you have that, you get to the final stage, phase four, which is publishing results, right? This is where you actually make money. If you don't get here, you haven't really made the business any money, which means you have an academic research project, not a product. And this is really, again, where I think Ruby is almost the perfect fit, right? So when it comes to publishing results, if you really wanna talk about a product, right? You're gonna end up building a web UI or a mobile app, right? So here's an example. That's ShareThru's analytics dashboard, right? So that whole thing is all Ruby. Because once you've done all that number crunching, all that modeling, you end up with actually really small data, right? And Rails is awesome at taking something from MySQL, from Postgres, and making it so that you can put it to the client, right? And once it's on the client, you have D3, right? You have high charts, all those things in JavaScript that we're all so used to using and we already know how to use. We just harness those, right? Like once you have the data distilled, once you've cleaned it, you've modeled it, and you're ready to present it, really. It's just building a web app or a mobile app, right? Here we have Yelp, right? Kinda the same thing. They've done tons of behind the scenes coding so that they can give you good recommendations filled throughout fraud. All of those things are already there, right? We already have that. And the reason we have that is Rails, right? So just because we're talking about building data products and we're talking about big data, doesn't mean we can't use Rails, right? When it comes to publishing your results, oftentimes you're gonna end up building dashboards, all those kind of things. Rails is great at them. So to kind of close my personal example, right? I said I needed to basically predict user overlap. So I'm not 100% done with that whole process, right? I've been having to iterate on those last two phases. I had a model. People didn't really like how they had to interact with it so we've gone back. But we get something cool like this, right? This is a little bit of eye candy I thought I'd put on the slide for you. And what this basically shows is a dense network of publisher overlap, right?  So kind of the nice thing, the thing I'm kind of trying to get across is even though you're building a data product, there's no reason we can't use Ruby. However, I have to note, unfortunately, I did not generate that graph with Ruby. I was a little bit dirty and I used Python. And that's because even though I love Ruby and I love the stuff you can do with it to build data products, it's not all roses yet, right? There's definitely things I think we as a community need to improve and come work on to make it so that we get more people talking about doing data things in Ruby. And kind of here are the four things I think we really need to work on. The first one is kind of a graphing library. I know there's been attempts, right? Like there's RubyViz attempting to port ProtoViz, but there's no really Ruby equivalent to the things people are used to in Python and R, right? In Python, you have matplotlib, in R, you have ggplot2, right? If Ruby could just have this centralized, adopted graphing library, it would go a long way because you spend a lot of time doing graphs when you're doing kind of exploratory analysis. And kind of the second thing is a unified matrix and vector library, right? Machine learning, if any of you guys do it, you pretty much know what it comes down to most of the time, it's just matrix math, right? You're like, you're trying to do some matrix transformation so that you can get like a matrix that you can work with, or you're trying to like fill in missing gaps in a matrix, right? All that kind of stuff. It'd be great if Ruby had kind of a more unified, centralized matrix library, right? The guys at CyRuby are working on it. I think there's a couple of other libraries out there if you look on GitHub, but there's no kind of centralized knowledge, which is really what we need, right? Like it's not that Ruby can't do it, it's really just that we need to like centralize discussion around it, right? Just like we have that centralized web framework, which made it so that everyone started doing web stuff and rails, and kind of rails to a certain extent helped us beat Python for web stuff, we kind of need that for Ruby. And really it comes down to those last two things, which is we just need more publishing, right? Like we need people talking about it more. We spend a lot of time in the Ruby community talking about like TDD and OO, and all those things are great, but it would be awesome if we started to get a little bit more publishing around kind of Ruby and machine learning and how we can really use Ruby. And that kind of ties into last one, which is academic buy-in, right? So like most of the time if you guys are going out to try and hire people to like build data products or do stats or be analysts, they're mostly academics, right? And academics tend to use Python and R, simply because there's not a lot of academic buy-in at least in California universities for Ruby. If we could just kind of get the academics to accept Ruby, see that we can use it to do kind of computer science-y things, because that's what computer science programs are all about, it would go a long way because people would come out into the workforce already knowing that they could use Ruby. So even though it has its words, I definitely think Ruby plus data equals agile data products, right? It gives you the ability to iterate really quickly. You can harness the power of Hadoop, you can collect data really easily, and then you can present the data that you've built using Ruby, using Rails, using Sinatra. And finally, my obligatory We're Hiring slide. So if you wanna work with me on data things, feel free to email me or visit that link. Thank you.