Alan, welcome to NewsClick. Today we are going to discuss the real face of Facebook. And I have with me two authors of the book, The Real Face of Facebook in India, and the real face of Facebook, which has been published in Hindi. Cyril Sammukh is the author of the book, and Poranjoy has helped Cyril with this book. So he has helped Cyril with this book. Of course Poranjoy, as you know, has been formed. And we thank you because NewsClick also published the first five long articles, which were published in English and Hindi. Cyril, you've been the lead author of this book. What led you to this topic? Because Facebook has been covered extensively abroad as well. It's one of the companies which has come under the scanner, and particularly after the Cambridge Analytica has been on, shall we say, the hot seat. Mark Zuckerberg has been called to different places. Of course he didn't come to India, so it didn't seem to work. And in England also he's been called, which he didn't respond. But did it appear in the US Congress? What led you to write this book in the sense that since already a lot of information has come out? A lot of information has come out about how Facebook has a technology or infrastructure functions, how certain kinds of things go viral on Facebook, how Facebook prefers a certain kind of content, etc., etc. But I don't think a lot of people have looked into Facebook's politics in different countries. We have very little information about Facebook's politics in different countries. We know that Facebook worked with the Trump campaign at some point. We now know that Facebook worked with the far right in Germany at some point. And in that I was curious what is happening with Facebook in India because the company is more or less opaque. We don't know what Facebook does in India. So I was just curious about what does Facebook's work in India, what are they doing in India, and that's how this came around. Your book goes into details about, shall we say, a revolving door of personnel. It also seems that Facebook has shared a lot of information which it doesn't normally do with outsiders with the BJP, maybe with other parties we don't know, but certainly with the BJP. Do you know for sure which country they have chosen or which political power they are going to be in? They are trying to create a relationship with them. If you look at the different countries, Cyril spoke about America, Europe, it happened in Argentina, in the Philippines, and this is also a very important tip for the United Nations, how Facebook and WhatsApp helped to kill Rohingya Muslims. So there is no reason for this to happen, that Facebook is trying to create its own profit.  Those who are in fact, they have more money. And while maximizing profits, what they end up doing is supporting majoritarian regimes, dictatorships, and there are a lot of examples which, like the politicians who believe in Tana Shahi, they got a lot of help. And what Cyril said, I would like to add a couple of things, that six months or eight months ago, when this work started around June 2018, at that time a lot of people were saying, how did Facebook help them? And let me give you one example, about Shivnath Thuklal, you said about the revolving door. Shivnath was a television anchor in NDTV, then he joined the SR group from there. Then in 2013, when the elections were coming, 2014 elections, he joined a website called Meera Barossa, which helped him to support Iqdam Modi. And then for a few days, he joined the Carnegie Foundation. And then Facebook, that in a way, after Aakhidas, he was in the number two position. So we asked Facebook this question, that when you made Shivnath Ji your employee, did you check his background? Yes, we have followed, we have due diligence, international best practices. We have also written that Shivnath Ji's partner, Shaili Ji, Shaili Chopla, he was also an anchor in NDTV, he also has a project, he has a YouTube channel and television program, She the People. So I said, Facebook, you are helping them, you are financially supporting She the People. Isn't this a conflict of interest? Their answer was that we have been supporting this project, even before Shivnath joined us. So, there are not many people who have all these thoughts. So that's why we, in the previous news click, published a book, and now in the book, we have said all these things to encourage people. How does Facebook really work, doesn't it really work? It is not a truly agnostic platform. It is not that any political party, or any individual can help them. In a way, those who are in power, and those who are worried about the South, helped a lot in the same way. And not just Facebook, WhatsApp. I think today, if around 22 crores, people around India are using Facebook, more than 30 crores are using WhatsApp. I mean, those who are giving money to India, around 22 crores, are using WhatsApp. And Malviya, the IT cell of the Indian People's Party, once in the economic times, when he was in power, said this is going to be India's WhatsApp election for the first time. You know, when you talk about the WhatsApp influence, it's also true that in the Brazil elections, the WhatsApp, the last seven days played a very important role. It's not documented.  And this was targeted towards women, particularly, and this led to a huge shift in the last seven days of the election. Otherwise, Bolsonaro was trailing till that point of time. Of course, all of these are publicly known in Brazil, but again, the Bolsonaro did win the elections. But leaving that out, that WhatsApp is going to play a huge role, the question still remains, and Seril wanted to ask you what is the reason that right-wing has a lot of money, that obviously business prefers right-wing politics to even, shall we say, social democratic politics or even welfare politics which should be considered bourgeois politics by any standards, that which argues that tax revenues should be such that the rich should be taxed in order to provide basic minimum education and health to the people that the right doesn't like this. It wants tax breaks and so on. And they have the money, clearly. And increasingly, the welfare structure has been weakened. So they have even more money than before. This has been, Piketty says, the most unequal society that we have seen for the last 100, 125 years. In this, do you see Facebook as a this exercise of supporting certain sections, in India particularly, especially, as it's ideological predilection, ideological wishes of its people, policy of the company or that profit maximization means it's a natural alliance or is it a matter of both? Do you have any opinions on this or objectively, what do you see? I think it's a mix of both because Facebook as a company is answerable to its shareholders and they want to increase their value. And it kind of couples with the right-wing's tactics because the right-wing has always been at the forefront, especially in India, they've been at the forefront of using technology. I mean, we've had the Vishwa Hindu Parish at the RSS using the Internet to mobilize people in the U.S., in Europe, even before Internet was a reality in India. And in that, yes, they've been far ahead of anyone and it's just a natural alliance. I mean, it just comes together. They were looking for a client, they found a convenient client in the BJP and they've worked together since. I want to add one or two points to what Cyril has said. Look, the age of the Internet is 27 years, the age of Facebook is 15 years. So in the last 15 years what you would call social media platforms, what kind of digital monopolies their use has increased. We never thought that it would grow so fast. It has grown so rapidly. And as we just talked about the benefit of this which is believed to be the biggest benefit for them. So the Internet we were thinking at one time that it's a universal good, it's a universal common which is like a public utility. Today we have reached a point where you are also aware that there are two major social media platforms Facebook, WhatsApp, Instagram and YouTube. These are the two digital monopolies of Maharashtra. The number of messages you are sending on the Internet is more than 70% Coming back to this I still have this interest that when we talk about what we have also written fake news that both WhatsApp Facebook partly also used to have become purveyors of fake news in a big way and a lot of it is due to also what people want to spread so it is sort of also feeding into that. So do you think that this needs to be controlled by regulation of some kind? Is it Facebook's internal regulation, external regulation or do you think the people by large as this phase becomes more and more shall we say past it becomes old, that people will become more and more educated and will stop believing a lot of this. So the problem essentially is with the architecture of these platforms these platforms are built to maximize time spent and engagement engagement is comments, likes shares etc. and that happens primarily and that architecture essentially prefers certain kind of content so say politics or crime or local news so that kind of content spreads a lot faster. So what we are seeing in fake news in India for example is that it is extremely hyper local so your local RWA guy is also producing narratives local panchayat leader is also producing narratives and it is happening at that level and these platforms therefore prefer to pick up those kinds of things and they try and push that kind of content. Will any kind of regulation help this? I don't know if any kind of regulation will help this because the entire commercial internet's economics today runs on engagement and in that it prefers sensationalist content it prefers things that look scary that will scare people so it is essentially Fox News or Republic TV transferred on the internet that is essentially what is happening here Regulation could be Elizabeth Warren saying she is going to break up all this Even then the fundamental architecture remains the same and the manner in which they make money but you know at the end of it there is no hope that would be a very unfortunate one So here is what the hope is and I think when we talk about all of this we assume the permanence of these companies every software has a life cycle of around 10 to 12 years and Facebook for example has been around for 15 years yes day before actually Facebook announced a whole bunch of updates to all of its platforms they are moving away from public to private Facebook says that the future of internet is privacy at some level Facebook understands that their game is over people are migrating to other platforms and in that this is an attempt the new updates are an attempt to consolidate what is left behind and in that give it another 5 years I don't think Facebook is going to be around ok there are two things one is that today there is a Facebook that is using around 40 to 33% of the world there must be some reason they are providing some services to communicate with each other 40 years ago we said to each other what happened to us all of a sudden one thing is to think and one is to not and when Facebook becomes viral they are making people emotionally react when you are in anger they like it when you are in anger then your post is going to be viral and if it becomes viral we think look at these two new words thanks to Facebook one friend that we have thousands of friends if our health is bad they will come to our house we will not help them to show this second thing is these digital media platforms social media platforms you understand that all these services are available to us for this digital platform you have become the product you have become a service that's ok we really do or have discussed this earlier also but yes, I apologize more discussion of what the disease is is what matter what is the solution to the disease this book is written by us and we have published it in Hindi we hope that in other languages we will be able to publish it in the future that one is awareness second thing second thing is because if people don't know the problem we have no hope of a solution now there is a reason as they have said when people's awareness will be filled the thinking that is coming you will not believe you can send anything so you talk about the education of the that is one the self education of the awareness together which is in the government in our country how do we control the internet and the social media let me give you a small example in Rajasthan a few months ago in Rajasthan one person what is the murder of another person and there is a video in their website so there was a worker he was killed and when the question was raised what is the murder what is the bad thing who made this whatsapp video who did it how did they do it and whatsapp said this is end to end encryption we don't know you can forget about that also look at the Christchurch shootings it was on facebook and for quite some time facebook failed to pull it down and already quote unquote viral by which time number of people had died by which time number of people had died but it went on propagating even after the incident was over facebook was aware of it and there is no end to end encryption in facebook so this is the problem of quote unquote virality and the fact that controls work much slower the internet changes the logic of publishing whereas initially you have filter and publish and then filter and that is also the reason why there is a lot of conversation around the loss of power of gatekeeping that editors, news organizations etc had because once everyone has access also because one of the first problems that the internet solved was the problem of publishing and distribution which was a huge bottleneck before the internet came around and we have made and we have continuously even today we are making progress in terms of how do we publish and distribute so facebook is one example, medium is another example blogspot at an earlier at an earlier point in time it was an example so the internet has solved that problem of publishing and distribution which media companies at one point had a monopoly over now publishing and distributing is the cost theoretically is zero I can sit and write something and put it up on the internet and ask me anything and I can ensure that it reaches the price you are paying for that internet but that logic of filter and publish I think that's turned on its head the second thing is any form of regulation if we go and regulate these companies right now what will happen is they are going to be around here forever you were saying it's better to have the natural life cycle of these companies rather than try and regulate them which will create how much of this you know the cycle, the march of technology the business cycle of technology will influence so Microsoft had a virtual monopoly on browsers today it's been overtaken by Google Chrome and Mozilla Firefox which are offering their services free similarly do you think that ok just because Netscape was killed by Microsoft and Microsoft got marginalized by Google Chrome and your Mozilla Firefox will be see a similar kind of thing will now a new avatar of Facebook or Instagram or WhatsApp come to occupy that position I am not so sure I think the jury is very open on that one because we are forgetting one thing that once you have say you are reaching a near trillion dollars size of a company your ability to buy new entrants is so much more today that competition becomes much more difficult so anybody who becomes a competitor WhatsApp was a very small competitor was bought over so unless they make stupid mistakes which Microsoft did when it came to the internet it is difficult to see how by natural competition life cycle of software which really is more hopeful of this will really happen if I can get a brief point in Zuckerberg himself has written in his article in his op-ed piece in the Washington Post and he is talking about privacy and freedom of expression I am saying that whatever Zuckerberg says don't take it so seriously it is a crisis of credibility Facebook and Zuckerberg is a credibility crisis therefore they are saying a lot of things in the trust of the people talk about privacy these are today's buzz words people are a little worried Standard oil is broken up AT&T was broken up maybe the time has to come to break up these companies or at least create a more even playing field through regulation Regulation necessarily does not have to be policing censorship controls of different kinds give us access and so on that with 1 billion people are producers of news today against 3 billion consumers by the way that is the ratio 1 is to 2, 1 is to 3 is the ratio then we have to think of new models of how do you regulate content no easy answers to this question if there was we would have given one earlier but I do think that your book raises very very important questions because if people do not know the disease if they do not know the disease then what kind of medicine they will not be able to cure how they will be able to cure so now in our opinion we are at the moment on the analysis of the disease and when it will be cured how it will be what to say to reach the beginning we will have to wait a little more and we hope that the treatment of this disease will be cured thank you this is all the time we have today do keep watching news click


Welcome back everyone. Last time yesterday, where did we leave everyone off? We left it off with some basic stuff. Some questions about the indexing, the lock and iLock. Yes, go back to that for a bit. That's very good. I think we skipped over that maybe a little bit too quick. Let's go back to that. So yeah, we had some good questions. So how exactly now do we select columns and rows? And the truth is there are many, many ways in pandas to do this. And if you try to take in all of these different methods at once, it will be a giant mess in your brain. So let's see if we can simplify it a little bit. To two cases. So maybe you can demonstrate. I know let's say a case number one is that we want to select columns by name. Just a quick reminder from yesterday, we had this Titanic passenger database that contains some information about each passenger. And let's use info to list columns. So this is all the data, all the columns of data that we have in the data frame. So what should I do? So when we select a column by name, then we use just the square brackets and we just give it the column name. Okay, that's how you select a column by name. So and now select two columns. So you can also give it a list of names, which will select multiple columns by name. So let's do the name. Brian, can you be a bit softer or Yarno a bit louder? I can be a bit softer. Okay, thanks. Okay, so let's try. Now I shouldn't be too much louder. Give it another column. Let's say it's survived. Yeah, so this is how you select multiple columns by name. The agent survived. You pass it a list of names. So when we're selecting columns by names, that's when you use just plain square brackets. If you're doing anything else that is not selecting columns by names, anything else, we're going to use either dot lock or dot I lock. So let's do lock first. So lock also allows us to select rows and columns by name. And so this is with lock you index it like you would index a 2d non pi array. So say we want row and then column first row and then column. And if you leave away the column part can also only select rows just like in so here is a row name. Yes, we do it by name. And this would now select a single row. Yes, let's try. Okay, so now if you want to select both a row and a column, I would do that. Okay, so first row then column. Yes. And for example, if you want all the column or a range of columns, so you want to slice it. Okay, range of columns. It's still by name, remember? Yeah, so we're slicing columns by name. Okay, so we'll take from name to age. Yeah, is name still one of our regular columns? Um, actually probably not. So age is here. Let's go from survived to age. That's how you do it. Yeah, so if you want to select both rows and columns or just rows, anything that's not just selecting column by name, you use lock. Now, if you want to use integer indexes, so we don't want to like write out these long strings, we want for example, we want row number 20 to 30. So instead of this whole name, I want to use a number. Yes. To select a particular person. Yes, or range of person. You said 20. Yeah. So dangling Ilock. And it will tell me the name fortunately. Okay. And Ilock works exactly the same as lock, but now you use integer numbers to identify rows and columns. But you can do the same thing. You can slice it up. Say we want rows 20, 25 and 30. We want the three random rows. You give that as a list. That's why. Yeah, that. Yeah. Okay. I was about to ask. So the first entry is the rows and the second entry is the columns. So it needs to be one thing. And in this case, it's a list. Yeah. So we get three things. And then say if you also want like the second and the third column or something. Yeah. Like this. Yeah. So second and third columns are passenger class insects. And I guess you could also do it by slicing like this. Yep. Actually slicing from two to four, because it's not inclusive of four. Okay. So I think with these use cases, I think you can do almost anything that you want. So to quickly recap, if you are selecting a column by name, then you use regular square brackets. That's it's only if you're selecting columns by name. If you do anything else that is not selecting a column by name, you use either lock if you want to specify stuff by name or use I lock if you want to specify stuff by number. And those are the three cases you need to know. And then you could do anything. Let's leave it at that. Okay. Okay. So let us know in the HackMD if it's now more clear. If you still have questions, put them there and we will try to get to them. But let's let's move onwards. So we've now up to now with, I think we've always now not always, but we've been loading our data using some of these read functions, like CSV. And we can also read from Excel sheets. We can read from HDF files, rotation files, anything we want. And pandas can write to all these file formats. But maybe we should quickly show how to create these data frames from ourselves within Python. Maybe if we have some data in NumPy, how do we get that into a data? Okay. So first we'll create a NumPy array. And to do that, I have to import NumPy. Okay. And then I guess we can create random data. So it's a quick reminder of the some things from the NumPy lesson. So I'm too many parenthesis. I'm creating a two times four matrix. Six times four, yeah. Six times four. What am I saying? Okay. And I should save it into a variable. Yeah, let's do that. Okay. So now, if we want to put that into a data frame. So new data frame. What would be the easiest ways to do it, like the shortest way? Hmm. Shortest way, not sure. But I probably want to construct a data frame. Shortest way would just be the past matrix and not do anything else. Now it will not give any names or well, it will not give any useful names to the rows or the columns. No, just not. They're just zero, one, two, three, zero, one, two, three, four, five. But the data's in there. So if we want to specify some columns, let's specify some column names. Okay. Specify column names. So columns equals and then a list. A list of column names, yeah. At this point, just to make sure that this fits into on the screen. I'll do this. Okay. Now, hmm. Maybe just call them A, B, C and D. I can call them anything, of course. Yeah. Right now, the content isn't really, doesn't make too much sense anyway. They're just random numbers. So yeah, let's call them A, B, C and D. Right. And now let's give the rows some names. Yeah. So, um, I guess I can do rows equals. Yeah, unfortunately, sorry. Pandas calls it something else. Yeah. Pandas calls this the index. Yeah. Um, so I could give it a list. Yes. Should we try that? In the materials, there's a somewhat different example. Maybe we could try that. Which is also nice to show. Maybe first show that you can just give it a list and it will do what you expected to do. Okay. Now I have six names. Yeah. So now the rows have names. Beautiful. Yeah. But the index, also the column names, by the way, but also the index, it doesn't have to be string names. Let's quickly take a look at another super useful thing. And that is to put dates there. So, okay. It can be, uh, uh, we have to define dates. Where each row corresponds to a different point in time. So we can put date objects there. So Pandas has a function called date range. Which will give me six different dates. And apparently I have to give it a starting point. So, um, let's do 2021. Um, that now is 10th and 26th. So that minus five is 21. Okay. And then I want six days. Now it's called periods because you can do other things than days. Right. Um, but by default it's a date, a period if it is a day. So this should give me six different days. Yes. Let's see if it works. It does not. Um, you call it data range. Ah, data. No, date, date range. Okay. All right. So now each row is a different day. Yeah. Ending at today. Beautiful. Right. All right. So, um, that's one way to do it. So now we have, so our original data was in the form of a numpy array. Um, if we have like different columns, like we have like one column with numbers and one column with text and one column with dates, maybe all kinds of different stuff. Let's do that. We can't easily put that into a numpy array. So usually we, we, um, so, so here we probably want to use a dictionary. We have the data and in dictionaries, I want to put it into a data frame. Okay. So we'll create a data frame again. What I want to do is this. Okay. And now, or maybe what I want to do is actually this. So I will start a dictionary here. Okay. Yeah. And what does this dictionary contain? Well, keys. The dictionary has keys like this and then it has entries for those keys. And I guess this will be, this will be rows. No, this will be columns. So the keys will be the columns. This will be the column names. Like here. So remember that a Python data frame, a pandas data frame is always a collection of columns. That's how we find it. So also our dictionary will be a collection of columns. So the key will be your column name and then the value will be the contents of the column. And the values will, okay. Values will be the contents of the column, which makes sense because this will actually be turned into numpy arrays. Yes. Or arrays of objects, but preferably numpy arrays in most cases. So if I put something like, um, should, maybe first some, some numbers or Yeah, I'm now reports are you're a bit louder than right? Okay, but not by much. Okay. So what I'm wondering is if I should go back to the previous exercise at this point and call these runners already. Oh, that's maybe actually a good idea. So we're now recreating one of the data frames from, from yesterday. Yeah. Actually, maybe just copy paste that. So now this is a list. That's another way to do it. You can either give it a list of dictionaries. Yeah, but it can be just one dictionary, but you can also give it one dictionary. So there are many ways of doing this. Again, just like with the indexing, there are many ways of doing this. So we had runner one, runner two, runner three, three, four. Now this doesn't quite fit. No, let's just do three runners. Just three of them. Oh, okay. Yeah, true. We had only three in the, in the example. Okay. So that's a runner column. And let's do another column. Maybe age, age of the runner. And that should be a list. So 21. I mean, these are, of course, professional runners. So they should be relatively young. It's 229. Yeah. I'm not sure if that's realistic anymore. Okay. Here we go. So we have three runners and we have defined ages for these runners. So that's another way to do it. Yeah. So actually now we've seen three ways. You can do it from a numpy array. And you can do it from dictionaries, either from a list of dictionaries or from a single dictionary. So it depends on what formats your data is in. Usually you can just give it to a pen as data frame and it will try to do the right thing. Let's see how we, all right. Let's go through some other things we will have to do. Like, for example, splitting and combining data sets. How did that work? So maybe first splitting. So maybe split this little table you now made into, like, only the first entry and then the last two entries. And let's be consistent. So we're going to deviate slightly from the dictionaries. Let's use the indexing we've been promoting at the start. So are we selecting columns by name in this case? Well, in the example, no. No. So we're going to use a lock or iLock. Yeah. Okay. And well, we can do both. Well, we can split it into two in one row. We could also do this in two rows. But let's take use lock. Let's use iLock. Yes. If you want to do a range, a numerical range. And this will get the first two rows. Yeah. That's fine. And then the second part will be the rest. So yeah, from two forward. That's great. So that should give you yeah, that should work, I think. Now it didn't print anything. But we have a table sub one, which is the first two rows from here. And a table sub two, which only contains the last row. So what if we want to glue them back together again? Okay. So if we have two data frames that we want to put together, and they have exactly the same columns. Yes, that's more convenient. Yeah. You can also do it when they have different columns. So we'll get to that. So I guess the most general option is to use concat concatenate. And that just takes a list of data, data frames. So sub one and sub two. And because I'm not assigning it to anything, it will print it. Okay. So we have our original pack back. Yeah. Okay. So that's yeah, splitting concatenating that you can do. But that's that's when the columns match. Okay, yeah, let's let's. So now we get to a point where where we would like to discuss merging these data frames where the columns don't match. And that is actually, in my opinion, one of the killer features of pandas. This is where pandas really starts to shine. So we now have this great little table of the runners and their age. And we have previously defined the table of the runners and the time it took for them to run a certain distance, which I have in fact overwritten. So can we quickly recreate that again? Yes. So here we created the database with data frame with these three runners and the times. And we use this melt operation to bring that into a canonical form, where each row has one time for one kind of measurement and the distance for where that measurement was taken. Okay. So we have now these many columns, each either runner one, runner two, or runner three and different variables for each. And we have a table that contains for each runner their age. And I will need to change that into something for the different names. So let's call it the ages database data frame. I keep saying database. Okay. So now we have data. Df is the big data frame. And then we have the ages in a smaller, yeah, smaller frame here. Yeah. Smaller table. This can be a, this is sort of a little example of a scenario where you pull data from different sources so that ages may come from the population back to history and the big table comes from the, from the running competition. So we now want to combine this data from these two different sources and want to just smoosh them together. And if we want to do this manually, it's going to be, well, it's going to be a bit of trouble because we have to go like row by row because nothing matches. The columns don't match. The rows don't match. We cannot just glue them together. We actually have to like piece everything together. But pandas can do it for us in really elegant way. And it's called a merge. So again, if you're familiar with database systems, this is what you do all day. You merge to get a different tables. Pandas can do the same. So there is a merge function. Yeah. So say for, we, yeah, we want, maybe use the, the big table as a basis. So maybe call it. Okay. So the big table comes first. Yeah. Well, well, let's, let's use the merge as a method of the first. Okay. Df.merge. Yes. Let's do it like that. So our Df, our big table is like the basis. And now we're going to merge in the ages of the runners. So this is now not the pd.merge, which would be a function inside pandas. Pandas is pd here. This is df.merge, which is basically the same function, but we are calling it, well, we're calling it from or with the Df data frame. Yes. The big data frame. And we'll have another data frame as an argument here. Yes. So, and that would be the ages data frame. And now there's one more thing we need to tell pandas. And that is what column to use to match the rows together. Yeah. So there's only, of course, here, one column that has the same values. So we cannot use age because the other table doesn't have ages and time or distance. But in principle, there might be two columns that match. Yes. But here we'll use the runner column. Yeah. So we're going to use the values in the runner column to see which rows should match to which rows. So let's see what this does. Okay. It has now successfully annotated our data with the ages of the runner. See in it, whenever runner one appears in the set, it will have, it has annotated with the correct age. So this is super useful for pulling together data from multiple sources and editing things. And so pandas is very clever. So you have merge, but you have, this is one way of merging. So the merge method, if you look at the documentation, it has many parameters, many different types of how to merge, what to do with missing data, how to deal with columns that are in one data set, but not the other data set and so forth. So what almost anything you want to do, like with smushing together different tables, you can even do it. And it will take care of all the little finicky things, make sure that the rows match and the right ages aside to the runner here and that sort of thing. So it says a huge amount of time. Okay. So let's talk about another super powerful function. So merge super powerful. Let's talk about group by. Okay. I need to refresh this page. We've updated this morning. Let's try. Yeah, that's better. All right. So we've just come, yeah, we just came from our merge example from the runners. Let's talk about group by. So that, let's get back to the Titanic data set. I like it. It's nice and dramatic. So you know, the nautical saying goes that women and children first, right? So let's explore whether they applied that to the Titanic. So if you were a woman, if you were a child, did you actually have a better chance of surviving the accident? So for this, we have to group the data. We have to split the data into different groups. You have to split first by sex where you mill. So what we did yesterday was use group by to group by. And we just had one column here. We had the survived column. Yeah, that's what we did yesterday. But now we want to look at the survival rate. I mean, I'm sorry, group by. So say we want to first split them up by, yeah. And then we can take the mean of. Now we want to select a column. We want to select a column by name. If we would do anything else, we would use a lock or iron. And then we do it. Then we take the mean. So this came to the survival rate. Mean of survived is the survival rate. And it will give it for each sex separately. So I think we did this yesterday or something very similar. But now we want to do something a bit more complicated. So we split up the data even more. That makes two groups.  Women and children first. But we don't have that data and that information. Yeah, we don't have a column yet for whether one is a child. But we do have an age column, don't we? So let's see, when are your child? Well, it was a long time ago.  So if people are putting people into lifeboats, what do they consider a child who looks like a child? Yeah. So first, we're creating a new column. We are taking a column by name, but this column doesn't really exist yet. We're assigning to it. So this is completely legal in Python. And what are we assigning to it? Well, we're taking the age. And we want to check that it's smaller than something. Let's say 12. Yeah. Okay. And I kind of want to maybe see now what type of column did we create here? Let's do actually titanic.info. And did I create it twice? Well, actually overwrites one, because it's the same name. Okay. So our... Okay, but let's first send us a discussion of group by because it's such a powerful function. Yeah, okay. Almost every time you spend us, I'm doing group by stuff. Yes. So now we have the child column. And we want to know the survival rate of adults and children. Yes. And the child column was now, it's a column made out of Boolean value. So either true or false. Oh, right. That's the type of the column here. Yeah. And that's fine. We can use that to group by. Okay. So now we can group by two things, right? We can first split based on male-female. Give it a list of column names. And then we group by whether they work or not. Pandas, sorry. Yeah, this is great. We want the survival again, whether they survive or not, and to get the rate for each. So first we group, group by, not just group. Okay. We group by two things. So we create four groups. We take the survival for each group and the mean. So this is taking the survival rate. And it'll do it for each group separately. Like, well, we kind of saw earlier already. I made a mistake. What mistake did I make? The column not found. Survival. Oh, it's called survive. Survived. True. Ah, there we go. Now it worked. Okay.  So it's certainly beneficial to be a woman if you want to survive the tectonic accident. Yeah. I mean, it is kind of children first, yes, in that children go before males. But it is women and children first in that order, I guess. Yes, first the women and then perhaps a couple of children. Yeah, adult male. Well, the difference between 75 percent and 59 percent is 60 percent. It's not huge in this small sample of people. Yeah. Okay. So that's group by. Hi. Okay. So, yeah, let's now we've talked long enough, I think. Yeah, let's give them an exercise. So go and play a little bit with this tectonic data set. We've shown you a few very powerful functions. Can you answer these sort of questions? Like, what different friendly sizes exist? There's a unique method there in panels you can use. Yeah, and then find out. Okay. So when you have the family sizes, find the largest family size and what are the names of the people in them. And you can also create histograms using the distribution of these family sizes and passengers and things like that. You have a button function where you saw also yesterday called East. And after this exercise, we'll look at a bit more plotting functions. Now I would invite everyone to start doing this. So, so we give our welcome back everyone. Yeah, welcome. So should we, I mean, we probably won't go through the rest of the material, but we can quickly showcase some of the most important things, which is plotting. I think. So, and we can just do this with the tectonic database in the, if you go through the materials later, you will create a new database with noble laureate data, but the tectonic database is fun anyway. So we already did this, we just did it here, we did this group buy and took this created the survival rate or calculated the survival rates for each, well, each sex and each, whether they're children or not. So, yeah, and maybe we can plot that in, well, at least one way, a couple of different ways, maybe. So here we want to take, I'm actually plotting the age in my example. Can we plot the survival rate? Well, you can try easily. We could create a new column that is the survival. So when you plot box plots, you generally want continuous values there. Yes or no. So it needs to be for each person. Okay, we'll just take the So in box plot, you group buy, you do essentially group buy, but you use the buy argument here. So let's use the child column and the sex columns, they need to be strings. Okay, so it's a list of strings. It will group by these two columns and then plot the age as a box plot for each. And here it is. So kind of unsurprisingly, children tend to be younger than adults. This is 12, is the cutoff point here. Cutoff point. And the women were slightly younger, maybe? Not significantly. Okay. Okay, for a final trick, we only have one trick left, right? Then we are out of time. So let's create a bar plot. So let's do something simple. Let's just say, okay, we have three passenger classes here, first class, second class, third class. And we also know the amount of money people paid for their tickets. So we're doing group buy. We have three classes. Yeah, so for the bar plots, we need to do this grouping buy. We have to do it ourselves. And this is true for many of the plotting functions. So it's a general thing that often when you want to plot stuff like the mean prize, somebody paid for a ticket, you first need to do this computation yourself and then feed it through the plot. So before calculating the mean, you need to group because otherwise it will take the mean for everyone. But yeah, let's take the fare for each class separately. Take the mean of that. And this will return numbers as you saw before. So it returns numbers. But we want to plot that. So is it just plot? Yeah, and then you said the kind of plot to bar. Some of the plots have dedicated functions to them like the box plot. And there are many different kinds of plot that go with the plot and then kind equals bar. Or I think you can also just do box plot or line plots or circle diagrams, things like that. But this does show you quickly and visually how much more expensive first class is compared to the other two classes. How close to each other actually second and third class? Yeah, it's not that bad. Okay, I think that's the time we had. So I hope you all got an overview of what you can do with pandas and how powerful it can be. There's many, many, many more things to learn and a good place. It also has good documentation. So I recommend everyone go read the documentation. Whenever you want to do some sort of transformation to your data, chances are very good that pandas has a function that does exactly as you want, if you can find it in the documentation. All right, let's leave it here. Let's go back to well, yeah, let's go for a break.


Welcome to the webinar on securely provisioning cloud resources using Kivarno and Frostline. Before we get started, just a quick blurb about me. I'm Ritesh Patel, VP Products and co-founder at Nirmata. Nirmata is a company behind a popular open-source policy engine called Kivarno and Nirmata provides policy and management and governance solutions around Kivarno. In terms of my background, I have around 20 plus years experience in enterprise software, I've worked in Motorola, Nortel, Trapeze, Brocade and also have an MBA from Berkeley as well as a master's in computer science from Michigan State University. So let's jump in. So as you all know, Kubernetes has become that de facto orchestrator for containers and there's a lot of surveys talking about the adoption of Kubernetes. The CNCF survey mentions about 96% of enterprises have either started using Kubernetes or are evaluating it and what's also happening as Kubernetes becomes ubiquitous is that enterprises are building platforms on Kubernetes. So platform engineering has one of the top trends as indicated by Gartner where platform teams are leveraging Kubernetes as that orchestration layer along with other components, could be open source, could be proprietary components to build internal developer platforms and then developers in those organizations are consuming the internal platform and this really helps developers focus on their applications, focus on their workflows and not really have to deal with the operations and the nitty-gritty details of keeping the infrastructure up and running, configuring Kubernetes securely, things of that nature. So that's really what's a key trend as we see it. There's a lot of literature, a lot of content out there on why platform engineering the benefits but we won't get into that. We'll focus more on the topic on hand. So as we know and as we've heard, Kubernetes is really effective, really helpful in managing applications, containers at scale but it's also very complex to secure and scale the platform itself. There's a lot of different surveys that have shown that this increase in security issues as 93% of users have reported at least one security incident, a lot of it is related to misconfigurations because Kubernetes uses a declarative way of defining applications as well as the infrastructure and any misconfigurations could potentially open up an attack vector. So that's really why Kubernetes has become very difficult to scale Kubernetes, especially as the demand grows and as the infrastructure itself grows. So Kubernetes becomes more complex to secure as one scales the infrastructure. Another challenge or another issue is as misconfigurations are prevalent in Kubernetes and one of the top issues, the cost of finding these misconfigurations tends to be high if these misconfigurations are found late in your development cycle. There's research that shows that misconfigurations are orders of magnitude costly to fix in production instead of finding them early in your coding or testing cycle and that itself is a huge challenge and something that enterprises are looking to address. So in the next few slides, we'll talk about how some of these issues can be solved using a combination of open source projects like cross-plane and Q1. So cross-plane is an open source project. It's a Kubernetes native provisioning engine for cloud resources. So it allows you to provision cloud resources or it could be other applications. It's actually very extensible and enables infrastructure as code. So basically, it defines a set of custom resource definitions that these controllers monitor and then you can use Kubernetes native YAML to describe the resources and once the YAML is applied to the cluster, the controllers will provision those resources based on the configuration described in the YAML. The benefit of this approach is that you can leverage full power of Kubernetes like policies, namespaces, real role-based access controls and so on. It also allows you to, it can actually monitor state of your resources and it can reconcile those resources from the cloud. So that gives you a view into the health or the status of your resources. Another benefit of using cross-plane is creating abstraction. So you may have, if you want to deploy multiple resources, you can create an abstraction to deploy multiple cloud resources with a single signal abstracted YAML. And now because this is done in a Kubernetes native manner, it also allows for security and compliance to be enforced at the API level, at the Kubernetes level without exposing that complexity to developers. So it really enables that self-service approach that enterprises are looking for. And the community, the cross-plane community has, you know, providers for popular clouds like AWS, UCP, Azure, so on. So for enabling developer self-service using cross-plane, kind of the way typically it's done is by creating compositions, cross-plane compositions to abstract all infrastructure requirements that are needed to provision a particular, you know, service or a particular kind of resource in the cloud, right? So for example, if you want to provision, you know, a cluster in the cloud, you may need, you know, to provision virtual VPN, private networks, maybe you need to provision subnet, security groups, all of that can be complexity, can be abstracted away from the developer and the developer can just focus on, you know, providing some basic information for the resource and all of the details can be handled through composition. So that makes it really help useful for enabling developer self-service for cloud resources. So next, let's look at Kivarno. So Kivarno is a policy enforcement, policy engine, and it can enforce policies in Kubernetes clusters. So Kivarno is a CNCF incubating project and it runs as an admission controller in cluster. So Kivarno policies are written as Kubernetes native YAML and can be stored or, you know, in Git or can be applied directly. And, you know, when these policies are applied, depending on how the policies are configured, Kivarno can validate, mutate, or generate configurations based on policies. You can also, these policies typically are used to prevent misconfigurations to ensure, you know, compliance with the enterprise standards or you know, industry standards. And these policies can also be used to automate developer workflows. Several examples in the community of policies that can do automation, for example, when a new namespace is created, you know, you can generate other resources that may be needed for that namespace. In the community, there's tons of examples of policies, you know, right from part security policies to workload security, RBAC, multi-tenancy, etc. So next, let's look at the Kivarno architecture. So Kivarno is a, it runs as an admission controller. So it, you know, when it is deployed, webhooks are configured, you know, validating and mutating webhooks are configured so that Kivarno can receive every request that's made to the API server. And then Kivarno has a collection of controllers which then act on the API, on that request, depending on the policy. So, you know, there's a controller that can apply policies to that, to the API request. There's a background controller which can keep checking if any of the policies have been you know, violated, you know, by any of the existing resources. So there's a lot of kind of checks that Kivarno does to ensure that the, all the resources are meet the policy requirements. And if they don't, then Kivarno reports policy violations in the policy reports. You can also specify exceptions, policy exceptions, that's a new capability that was added to Kivarno recently. In case you want a certain resources, certain namespaces to be kind of, you know, excluded from these policies, policy exceptions can be used. But overall, the architecture is designed to handle the load that, you know, in very large clusters. So it's highly distributed, highly available, the separate controllers performing separate tasks. So if one controller, you know, has issues, it doesn't impact other controllers. And then the common reporting layer provides the ability to kind of get the reports in your cluster and then throw other tools, you know, make those reports available outside the cluster as well. So as you saw, both Kivarno and Crossplane are Kubernetes native tools. And one Crossplane is for infrastructure as code for provisioning resources, Kivarno is for policy as code. And there's several benefits of using Kubernetes native approaches in your platform, you know, for developer self-service, for, you know, automation, and so on, right? So one of the benefits is obviously they are, since they are Kubernetes native, they're using, they let the Kubernetes API machinery, a lot of the resources are defined as using CRDs. So as a Kubernetes user, the learning curve is very low, right? So you can use familiar tools, you can use Kubekartl, you can use Helm, you can use GitOps to deploy some of these YAMLs. Also, out of the box is integration with external projects. For example, there's endpoints for collecting Prometheus metrics. There is ability to use, you know, Argo CD for deploying some of these policies or other cloud resources. And then, you know, both Kivarno and Crossplane project have huge communities, as well as an ecosystem that helps if you need any help, you know, or run into any issues. So community can help with, you know, issues. Community also builds, you know, tools, adapters, providers, extensions around these core projects. And then there are commercial companies supporting these projects, like Nirmata for Kivarno, like a bond for Crossplane that can also help if needed, right? So there's a lot of benefits to using Kubernetes native approach when building developer platforms. So now let's focus on the use case that we were, we initially started the discussion with, right? So self-service cloud provisioning. So we'll have a demo of this use case a little bit later, but before that, let's look at how through using Crossplane developer can provision a cloud resource, right? So here I have a management cluster where Crossplane is installed. It's a Kubernetes cluster. You know, I can see the API server as well as the database. And then there's on the right, you have your, you know, cloud services. In this case, we look at AWS. And then the developer, let's say in this case, developer wants to create an S3 bucket. So the developer would specify the YAML to create the S3 bucket. We'll see what it looks like, but it's essentially a custom resource definition in the cluster configured by, created by Crossplane controllers. And then when the developer creates the S3 bucket, you know, the Crossplane controller and the AWS provider that's part of the Crossplane, one of the Crossplane components will actually detect the creation of the custom resource and it will trigger the creation of S3 bucket in the, using the AWS API, right? So that's kind of pretty straightforward. The developer applies the YAML and the controller, the Crossplane, the provider actually picks it up and creates the S3 bucket. It actually even, you know, periodically synchronizes with the configuration in the cloud and the developer will, you know, by looking at the custom resource, the developer would know if the bucket has been created or not, right? In case of any errors, the developer would know that, you know, would see the error in the CLI. So next, now the challenge with this is obviously, you know, any configuration, any cloud configuration has a lot of different knobs, a lot of different configuration parameters that have to be, you know, approved, checked, verified to ensure that these, you know, the configuration is not insecure. For example, the S3 bucket being created is not public. The S3 bucket, you know, information or any operation is being logged. There's a whole bunch of things that a platform engineer may want to do or there may be security requirements, you know, to be enforced. So for that, now this is where Kivorno would come in and help with enforcing some of these, or setting some of these guardrails as well as enforcing some policies, right? So in this case, now the kind of the model changes a little bit. You have a cluster, management cluster with crossplane and Kivorno both installed. And then the platform engineer can create these policies beforehand. We add these policies into the cluster, and then the Kivorno controllers will start monitoring the API server requests. And anytime any resources created that match the policies, it will apply the policies and allow or deny those requests depending on how the policies are configured, right? So once the policies are configured, now the developer workflow remains the same. But before the crossplane controller can act on those custom resources, in this case, creating the S3 bucket, Kivorno can intercept and apply the policies. And then if the policies, all the policies are successful, successfully applied, and there are no issues, then the request is completed, and then the crossplane controller can actually start processing the request and creating the S3 bucket. So that's kind of the flow. It's by adding these policies, now you have been able to set the guardrails and enforce various checks as needed in the developer workflow. And all of this without having developer having to learn anything, any policy violations will actually be available to the developer directly. They don't have to learn or know about Kivorno at all. So let's jump into the demo. So I have a cluster that is already configured. And it has Kivorno, it has crossplane already deployed. So before we start actually verifying the policies and seeing how it works, let's look at the policies and the configuration, right? So I have here, I have a set of policies for S3. And these policies are written in YAML, they operate on this bucket resource, which is a custom resource defined by the AWS provider, the crossplane AWS provider. And there's various checks, this policy checks for version, this policy checks if the S3 bucket is configured to access for public access. We have a check here for region, another check here for if notification is enabled, and then we have a check for logging. And then I have two different, you know, S3 buckets and I'll try to create one in USEs too, and another one in US West one. And we'll see how these, whether these get created and what happens, right? So let's go back to the command line. And I'm in the S3 policies repo to apply these policies, just going to apply all the policies in my folder. So it'll take a couple of seconds for these policies to be applied. Once the policies are applied, you can actually see these policies and make sure that they are ready. So here, all of these policies are ready. And you can see which some of them are in enforced mode, others are in audit mode. So now let me go to the resource folder and let me try to apply the policy that or the YAML to create S3 bucket in the East region. Actually, before I do that, let me show you what's running in the crossplane namespace. So here you can see the crossplane system namespace. We have the crossplane components and then we have a provider for the AWS cloud. And and if I do get CRDs and grab on crossplane, you will see all the CRDs that are installed in my cluster and a lot of all of these are most of these are related to the AWS provider, right? So these are created by AWS provider, these map to AWS resources. So for example, the one of the custom resource that is available here is the bucket resource. So we'll be creating that resource in the AWS in our AWS account. So let's let's go ahead and do that. All right. So this so when I try to create this resource, I get some violations, right? So because I'm trying to create this resource in US East, I have a policy which prevents me from doing that, which is this policy. And then there are some other policy violations that are also that are also triggered because my bucket is not configured correctly, right? The configuration for my bucket does not meet the policy requirements. So that's kind of how, you know, immediately developer gets feedback that, you know, they cannot configure this S3 bucket. So let's let's now try to configure the other S3 bucket in the US West region, right? So let's see if this works. So yeah, in this case, the S3 bucket got created. And if I do, if I check buckets, and I should see that my S3 bucket, the class frame S3 bucket is created. So maybe I just to verify, I can go to the AWS account and in my account, I can see my demo West S3 demo West bucket as created. So that kind of shows you now with this, the S3 bucket got created, it had, it pretty much had all the configuration that was correct. We can actually see if there are any policy violations. And in this case, we actually see that four of these passed, but then there was one audit violation which failed, which was around the email notifications being enabled. So it failed and it actually, but since it was set in audit mode, we could go ahead and create the S3 bucket, right? So if you enforce, policies in enforce mode will prevent the creation of the bucket. But if it's an audit mode, the resource gets created, but then you have a policy violation. And you can actually look at the policy violation in more detail by just using this command. So it tells you that this message tells you what happened and why it failed. Great, now the other aspect about using crossplain is you can also use crossplain to actually discover or find resources from your cloud account, right? So crossplain has the ability to adopt resources and you can do that. And in that case, and you can still apply policies, even though those resources are already created in your cloud account, you can apply policies and make sure that there are no policy violations, right? So what we'll do here is we'll go to our, we'll go to the elastic ECS account and just check in our ECS account. We have some task definitions and we'll adopt one of those task definitions and try and make sure that we are able to validate it and in case it has any, and check if it has any issues, right? So we have this ECS task definition created in my ECS account. So in order to do that, let's go back to the ECS folder and what we'll first do is we will apply these policies. So I have, so let's go look at the policies first. So here in the ECS account, for ECS, we have two policies to check if the image is coming from specific registry. And then we also have policy to check the task definition and to make sure that the CPU and memory are, you know, configured within the specified limits, right? So, and I know I've purposely kept the limit low so that we can, some policy violations can be triggered. So first, let's apply these policies. So if I do, I'll get C pull. I should see that I have now two policies and both of them are in audit mode because here we just want to detect we are not going to enforce anything because the resources are already created. So let's go to the resource folder. And here I'm going to just import the task, the task definition. So before we do that, let's look at the task definition YAML. So this is a very basic YAML. It just specifies the name of the task, the region, and only the required fields, right? So here we are going to just specify the required fields. And because it has this annotation called crossplane external name, it's actually going to just pull down the configuration of the task and add it in the custom resource. And also we've created configured the deletion policy is orphan, which means when this task definition is deleted from the cluster, it won't actually delete the class definition from the cloud. And this, this can be configured either way. But in this case, I've just done it to make sure that the original task definition is not deleted. So Google create minus F import task definition. So now let's look at the object itself. All right, so we have this task definition, and we can actually get more details. If you go to the YAML format, it's still synchronizing and it's getting additional information from the cluster. So this could take a few minutes. So let's maybe check again. Okay, so now we see all of the configuration for the task has been pulled down. There's a lot more information here. And so now let's see if there are any policy violations. All right, so we see two, we see one failure and one couple of errors. But in case of this failure, let's look at the failure. And this failure, this policy violation is for image check. So because the image for the task is not coming from the registries that we would like, this violation is triggered. And it says all the image images for this task definition should come from an authorized repository. So this is a way to actually apply policies, even to existing resources by using crossplane and Kverno, crossplane will actually synchronize those resources and then Kverno will apply those policies. So just to summarize, as you saw in the demo, you can use Kverno along with crossplane to enable secure self-service for developers. You can set the necessary guardrails when developers are provisioning cloud resources using crossplane. Kverno can provide the necessary policy enforcement, audit compliance for your cloud resources. And then the benefit of doing this proactively while provisioning these resources is you can pretty much prevent misconfigurations. You can overall improve security posture. And by preventing these misconfigurations and not having to deal with them later on in your production environments, you effectively save costs as well. So that's kind of the overall benefit. And this in the end leads to a much better developer experience without any back and forth without any friction that may typically be involved in such scenarios. So that's it for this webinar. Please give us some feedback, reach out to us if you have questions and would love to hear from you if this was helpful or if there are any other topics that you would like to hear from us. Thank you.