OK, thank you very much. I'll be talking about, not about OER this morning, normally I do, but today I will be talking about initiative taken by the Berlin Senate a couple of years ago at the commemoration of the falling walls, of the falling of the wall, of the Berlin wall in 2000 and of 1989. In 2019 a foundation has been founded calling falling walls and its aim was, and still is, to break walls, tear walls down in science and knowledge. There are several domains and one of the newest domains is learning and education. It started two years ago with the outbreak of the COVID-19 pandemic. And this falling walls is a yearly conference around the 9th of November. I was asked to make an assessment of the first and second round of nominations and I will tell you about these two kinds of nominations and it's central theme is accessibility, more access and more accessibility of education and learning. I'm struggling. And the central theme, what I said, is which are the next walls to fall, it also holds for education. And the falling walls initiative of learning is called future learning and the way it is organized it consists of three parts. It's a future lab that identifies supports and gives visibility to key initiatives. The second one is a hub for peer learning, networking and the third one is a competition that scans the horizon and I will be talking especially about a certain element of this competition. Each year falling walls for its ten areas, a competition is open. It's open at this very moment and the nominations for learning are being validated on four elements. First of all, the accessibility, how does it improve or enlarge the accessibility of learning and education? The second one is, is it sustainable? Is the initiative sustainable? What are the proofs of the sustainability? The third one is the impact, how much impact does the initiative have? And the fourth one is, is it scalable? Can it be replicated in different contexts in different situations? The network consists of more than 200 stakeholders and innovators and here are the figures for the 21 submission. There were 71 applicants, 24 were shortlisted by an international jury, a peer reviewed jury and from these ten winners resulted and then finally one will be selected by an international jury, another jury, as the breakthrough of the year. The applicants came from different countries. This is an overview of the countries. In total there were 71 submissions from 34 countries. In 2020 it was 35 countries. Europe is 37% of the submissions. In 2020 it was 41. Africa is 18%. In 2020 it was 7%. And North America is 15% and it was in 2020 32%. So there is more, there's a change in that the participation from the global south is increasing and this is the result of enlarging the network of peers from the various regions in the global south. The 71 submissions we categorized them using the whole long IQ taxonomy. It is a taxonomy originally used for allocating VC, this ad tech initiatives and this taxonomy gives an overview of the different elements which can be discerned with regard to learning and education. If you want to know more there will be a publication from the Fallen Walls Future Learning Initiative, one of these coming days. And then you can have a more close look at this taxonomy and how we derived it and enlarged it. In order to make an assessment of the impact of the submissions we used the SAMR topology. Yesterday it was also already mentioned also. And it says from four categories are being discerned. The first one is substitution. Is the innovation a direct substitute with no functional change? The second one is augmentation. Does it add something so that the function is slightly altered? The third one is the modification. Does the innovation allow significant changes? And the fourth one is a redefinition. Is it completely new? Does it redefine a specific task or activity of learning? And what we did is that we allocated the several initiatives and it turns out that 60% of them is of the 71. So more than half is in the category of modification. In fact 42 of the 71 entries. And 22 of these entries relate to important changes in learning. Apps, bootcamps, AR, VR, gaming and simulation. And new ways of learning STEM and coding. In the next slide which is difficult, oh no it's big enough, these are the 10 winners categories. And what you can see is they come from 10 different countries. So these initiatives were not selected on the basis of countries. They were selected on the information available on scaling impact and all these elements. And then peer review jury has selected these for 10 winners. And the last one is Rory. I will tell more about it. It is the breakthrough of the year. It comes from Ghana. It is developed by a for-profit organization from Canada. And it is a concerned tutoring. And on the last column you see the type of SAMR category. Most of them are modifications. And two of them are redefinitions. We will ask, we is Dominique Orr and I will ask to make some reflections about the type of entries and what can be said of possible developments. And what we said and we discerned the four trends. It is there is an emphasis on low-tech and low-tech used for scaling up and reaching impact. Secondly, there is an emphasis on networks which build new social relationships. The third one is within the submissions. There was an emphasis on learning labs, hybrid learning experiences to foster innovation. And the fourth one was recognizing supporting the power of learning in the wild. I will go into each of these trends and giving an example. This is the winner of last year. It is low-tech. It is George Cowell, the director of Rising Academy Network. And he developed with his people a low-cost tool used in the telephone to an audio tool for engaging a personalized learning to any student on any phone. He did it in Liberia, Ghana and Sierra Leone. And more than 50,000 people are being reached momentarily. And it entails a very simple tool. On each phone, a student can follow, can make a test, and then he has to answer a question. And then he or she gets a feedback whether it's a correct answer or a wrong answer. And also it gives directions where the pupil has to give attention to. It's a low-tech, the phone on the backbone, a very intelligent AI system. And it is a combination using low-tech with a high, sorry, with a high impact. The third one of the second one is how networks will help build new social relationships. It is an ad-tech platform with the mission of disrupting traditional secondary education. And it is a real-world learning as a core part of school. It is in place. It makes it easier for schools to reduce the opportunity gaps of their students by providing access to mentors for internships. The third element is learning labs. It is an ICT academy and Tech Hub providing its communities with the right range of technology platform on which people can learn and practice software development. It is an introduction of online jobs, allowing people to solve their problems. And it is being developed in Malawi and it is being used by both refugees and rural communities. And it is spread out momentarily through Africa as a whole. The fourth one is recognizing and supporting the power of learning in the wild. And this is by a sunny young M-school. It is a wandering cellist. It encourages students to step out of school, make the city their classroom and create meaningful learning and growth through interactions with the real world. And it is breaking the walls of the traditional schools by using also high-tech of a combination of low-tech and in its backbone high-tech. Lastly, we were also asked to reflect, to make a reflection on what is missing in the 71 submissions. And we found out to our opinion, three elements are just lacking and the emphasis will be laid upon the submissions for this year. It is the question of data, data mining and using data. And data privacy and security. There was no project in which this item appeared as a focus of attention. Secondly, even if sharing is caring, it is also a way to provide more inclusive digital learning for all. This element of more inclusive digital learning was also lacking as an element of attention and focus. And the last one is greening tech. It is also about securing a better learning future for all. This element was not in 2020 an element and not in 2021 a focus. There is still time to nominate your own project. This is the URL. And so if you are aware of a project, of yourself, of somebody you know, please submit it. Okay. And I think that we have some time. If anyone has any questions, perhaps I'll start off and just refer to one of these amazing projects. This is astonishing work. I actually typed something out so it would be a bit structured. So you spoke about an emphasis on networks and building relationships. I speak in particular about Rory AI in Sierra Leone. And you also referred to the low tech devices. So as someone from the global south, what are the bare minimum characteristics of such a low tech device if you are aware that would enable Rory to be functional? Just a very simple phone. A very simple telephone. So you don't need an iPad. You don't need an iPhone. Just old school telephones are being used. It has to have a screen. That's all. But because it is audio, the phone is just the phone. A very ordinary, old fashioned phone can be the instrument at the user side. On the backbone is very sophisticated. It's AI and all kinds of services. So Ben, thank you. With all these projects you mentioned, like you were saying, revolve around network building. How much of that is reliant on proprietary software for one? And for second, do you feel like if you would look more at European and North American context, people would be less inclined to use free or open software because there's better accessibility to proprietary software or better affordability of proprietary software for network building? Okay. I don't have an exact figure of the proportion of the projects submitted relying on proprietary software or open source. But I know that, let's say 64, 65% of the projects are projects from non-governmental organizations, not for profit organizations. And most of them use open source software or at least make the tools they develop available as open source tools. There are second part of the question whether it's more easy to rely on proprietary software. I don't know. I cannot give you the answer on this question. But it's a good question. I think we should include it in next year's evaluation. Hi. Thank you for the talk. My name is Will. I lead Frontiers for Young Minds in one of the sponsors of this event. We're an open access journal for kids. So there's quite a lot of synergy here. And we actually did put in an application ourselves back in April. My question is related to shaping content tracks in the direction of them. We publish individual articles, but we're looking to make sure that those articles that are written by real researchers can be integrated into local, national, or supranational curricula. My question would be in your view, would shaping mini curricula enhance our scalability for a classroom? Because at the moment, whilst we can publish individual articles written by researchers and reviewed by kids, and we can certainly collect them and publish them and collate them in collections, there's a tag missing on them, which is the kind of the pedagogical element to it. I'm curious to know if that's something that you think would help with scalability in those kinds of issues. It's a multifaceted question. You're asking about the size of the learning objects. What I know about the projects, it differs very much. Most of them are very small. There's not one initiative. There are some initiatives of the 71 which relate to a curriculum or part of a curriculum in its entire form. But most of them use very small learning objects. And your second part was, yeah? So the second part was around scalability for classrooms specifically. Because of course, whilst, for example, in your organization where open education resources can help with supplementing, whilst national curricular changes, etc. I'm curious to know if there's specific elements of OER resources that make them more suitable for scalability in those scenarios? We haven't looked at this particular aspect of scalability. We looked at the scalability of the initiatives in total in its entirety. And you are asking about one of the building blocks of the several initiatives. We haven't looked at it. I cannot give you an answer on your question. I'm sorry. Well, thank you. I think we'll move on to our second presentation. Thank you, Ben. Can we have a round of applause?


My name is John Kulshaw, I'm the Jack B. King University Librarian at the University of Iowa Libraries, and it's my privilege to serve as President of the Association of Research Libraries in 2021. Over the last seven months, ARL has intentionally and strategically crafted an action plan that will guide us in supporting you, our members over the next two years. Above all, this plan will assist the Association in supporting research libraries toward meeting the current, future, and fluctuating needs of the research and learning community. In spite of uncertainties and a different future, we know research library leaders are dedicated to advancing and exploring new pathways. ARL is here to help. COVID-19 presents both challenges and opportunities. We also recognize and prioritize the necessity of increasing diversity, equity, inclusion, and accessibility. ARL will support research library leaders as they address deep systemic and structural changes. In the course of our work, research libraries and archives are focused on sustained and strategic investments that build infrastructures. ARL can help you emphasize that the expertise and services of our libraries and archives directly benefits researchers, faculty, students, and others. We also demonstrate with data and evidence that research libraries and archives are positively impacting the research and learning ecosystem as well as policy and practice partners. Through the five objectives outlined in ARL's new action plan, we will further and strengthen the work and understanding of research libraries and implement impactful change in the research enterprise and overall society. These five objectives build on ARL's existing strengths. We will advocate and advance law and policy for balanced copyright, enhance partnerships toward open science, advance sustainable and systemic change, particularly as it relates to racism and social justice, implement a research agenda addressing impact and outcomes, and implement a strategy for leadership and organizational development for library deans, directors, and staff in our member libraries and beyond. In addition to these five objectives, there are two underpinning core commitments in the action plan. ARL will be welcoming in all activities and services, and we will collectively develop a set of plausible futures for research libraries and archives. As your new ARL president, and on behalf of the board and our association committees, we look forward to working with you. Thank you for your participation in the collaborative effort that led to the creation and adoption of the new action plan. Take a look at the action plan to learn more and also watch for Mary Lee Kennedy's upcoming blog post for more details. We welcome your continued feedback and engagement.


Good afternoon ladies and gentlemen. India is the largest democracy in the world. Actually India is the largest democracy in the history of the world. In May 2019 when the election took place, for the first time, almost 900 million human beings had the right to vote for electing a leader of a nation. This has never happened before in the history of this world. And in the world of democracy, India is the best student. We say what we want to say, when we want to say it and how we want to say it. The gentleman before us was speaking about how it took so long to build the bridge in Bombay, the sea link. And it takes minutes in China to do something. Well ladies and gentlemen, China does not allow its citizens to vote. We vote regularly in this country. It's a huge achievement for a country of 1.3 billion people. And in this democracy, one of the best things is the media. We have all kinds of media, we have all kinds of anchors, we have all kinds of channels. And 24-7, they take a position. They tell us what is happening, what we need to know. And so many of our opinions are based on this. Since the Pulwama attack and the Balakot strike and the 2019 election and now the upcoming visit of the US President Donald Trump to India. Television has yet again evolved to a different level, especially television news. I must thank the business world of having given me the opportunity to present before you some of the finest news anchors in this country who are normally not panellists in their own debates. So here I have a chance to grill them like they grill the other debaters. We are going to discuss a very important issue today and that issue is objectivity in news television. Does news television at some stage become propaganda? Does it become a tool for a political party to send its message? And even more, you know, going beyond that. What is the role of a news editor or a news anchor today to take a side or to remain consistent with the editorial policy of a network? We have had a preliminary discussion and we have very strong opinions on all of these subjects. But I'm going to start with the only lady today on the panel, Ms. Palke. You've had experience in multiple channels and now you're running beyond. Where do you think is Indian news networks as far as objectivity is concerned? I think objectivity is in short supply. I have to admit in the Indian media or media anywhere in the world. With the changing dynamics, with news and information, there's a glut of information. There was a time when it was said that if you don't read the newspapers, you're uninformed. If you read them, you're misinformed. Today, information is available online. And if we speak only about television news, I think people don't come to you necessarily for information. They come to you for validation. And so every channel has devised or every anchor has devised for himself or herself a position that he or she takes. And then people come because people want to hear what they believe in. And so, yes, you could say that objectivity is not always respected. But at the same time, I believe that the viewers have the ability to differentiate what is objective and what is propaganda. And beyond a point, all news tends to become propaganda. It depends on what headline you pick, how you phrase it, what adjectives you use. So it's, I don't want to paint this with broad strokes, but I do want to say that yes, objectivity can sometimes be compromised to take a stand. Mr. Rohit, what's going on? Some networks are propaganda, some are still a little objective. I'm not here to insult anyone.  No, but you have to answer the question. I'll tell you later. Please say it later, nation wants to know or not. So I'll read it later. I'll tell you a small story. Usually, when I interact with viewers, I interact with stories so that things get easier. There was an elephant in a village. And the villagers saw the elephant. They were talking about the elephant. We saw the elephant. There were five people in the village who didn't have eyes. They were blind. They also wanted us to see the elephant. So how would they see it?  In some people, he said it looked like a pipe. In others, he said it looked like a bush covered by a tree. In some people, the second leg looked like a worm. So they argued over this and that. So, a good man like you was talking to people and he was like, why are you fighting? We saw the elephant and we felt the elephant in pain.  But the point is that you are all looking at him in a different way. Bring everyone together. When you look at them together, you will come to know how the elephant is. So what is the work of a news channel? Is it the work of a news channel or is it the work of a news presenter? That there is also a fact, there is also a fact, there is also a fact. He has brought the sound, he has brought the pitch, he has brought the tongue. He has brought everything. He has brought the voice and kept it in front of the audience. So he will be called objective. And if any of them miss a part of it, then the other people who have seen any part in the other corner will say, no sir, this is not an objective, this is a lie. It cannot be a lie on its own. It is not necessary. Nowadays, in the newsroom, the word they have used is a dynamism. One thing is missing from you, it is somewhere in the other corner. It can happen that you will get it after a little while. But in that little while, if you have made a hasty decision to make, then people will say that you are not objective. You are on one side. But there is one thing, you are talking about elephant. Some people say that elephant was not a tiger. Sir, for them, I cannot say anything. If you tell elephant a tiger. No, but this is also happening. Sir, I feel that this is a little elephant that you are saying. You should call elephant a tiger. They will say that elephants can raise their weight a little. They can raise their small weight. But they will say that elephants can raise their weight a little. They can raise their small weight. But they will say that elephant. They will share the elephant. Sir, this will be unfair.  Mr. Bopayant. I feel that this subjectivity debate, which you should not only connect with the media, if you look at it in a social context, then I believe that there is nothing called objectivity anymore. Anymore. Was there objectivity maybe 15 years back? Maybe there was, maybe there wasn't. I think that what has happened is that with this advent of social media, I will tell you a story. At the time, I am talking about 10 years back, when Twitter was beginning to enter into news rooms. So our morning news meeting, our editor started to see that he was getting angry and angrier by the time he would come to the meeting. So he started asking me, boss, what's happening? Why are you losing your cool? This is going on on Twitter. This is going on. This trend is going on. Now, in the pre-Twitter era, the same editor would come. I would think in terms of, there are 10 reporters, what is their story? What is not happening in the whole country? So what has fundamentally changed, which has led to this debate between is there objectivity or is there no objectivity, is that social media has taken complete control. Hashtags. Now, the moderators have said many times that our hashtag is this. Now, there are so many anchors sitting here, it doesn't matter. Hashtags doesn't matter. So the editorial that used to be in the newspapers, now those are the basic news. What has happened? And I am sure you will bring this subject up later. What has happened is that the business models, the financial models of conventional news platforms, and it's applicable whether it's print or it's television, journalism today, real journalism, in terms of uncovering big stories, in terms of doing final scrutiny of data, which I think is the biggest problem. We just don't know which data to believe today. The government suppresses some data, or the data that comes up, there are many questions. There are some online platforms that are doing scrutiny. At least I don't remember whether there is a platform in TV channels or a platform in big newspapers, where you can scrutinize the data of the government. And the reason for that is that at least across the board, I don't see investments being made in terms of reporting talent. I don't see investments being made in letting people just go and do stories. In 2002, I remember, I was five years younger, and I was in a different organization. I was given an ambassador car and a cameraman. For one full month, there were no U.P. elections. There was no Obi-Wan. There was a center in Lucknow, and there used to be resets in that time. They started something from a private company. I would just travel because I did not have this pressure on my head, that I have to deliver something by the end of the day. Those days are now gone, long gone. There is just no money to be spent really on an extensive, massive ground report when it is possible. This is why the objectivity is ending. But you are competing with Twitter at one level, and the business model at another level. Just look at our friends here. We promote our program on Twitter more than anyone else. This is my hashtag. When I take a meeting with my team, our discussion is what should be the hashtag. What should be the hashtag of our coverage. That is the discussion. You are not the only one. The U.S. president also does that. This is why I am saying that this debate therefore needs to look at why it has happened. There is another challenge. Let me show you something on TV. There is a WhatsApp university. And the WhatsApp university is telling you something else. It has rewritten the history of India. Now many people have graduated and done a PhD in that university. How will you deal with that? That is fake news, really. That is the challenge. The real problem is that the ability to do a finite scrutiny of data today is impossible. Say when the government, when the party in power will say that I have created a number of jobs, but a member of Prime Minister's Economic Advisory Council will come out and say that this data which will give you an accurate picture is being suppressed by the government. So how do you come to a conclusion? Either you make it an agenda and run it. It doesn't work. We can discuss that as well. My real problem today with our industry is that I just don't see anywhere. I just don't see hard reportage taking place. I don't see any attempt really being made to get hold of stories which we do not know of. What are we selling? We will do some things here and there. We keep doing that. But this type of reportage, the fake news you are talking about, is very difficult. So, Mr. Sumit, you agree with that? Thank you. I would like to say two or three things. The discussion so far. First, you talked about WhatsApp University. My objection is that you should at least sit on this bench and talk about it. I am saying that. If it was a university, it was a cut-axe. It should be cut-axe. So, we didn't call it a university. You asked how the WhatsApp University runs on so-called things. We can save it from that. I have a crude idea. Because we are all worried about WhatsApp. We are more worried than happy. Earlier, we were happy. I think we all will agree with that. I am telling you my opinion without all the other panelists' wishes. WhatsApp should be paid. The first thing. It shouldn't be free. You are writing your own message. You are sharing it on Facebook. No, I don't have any problem. I want there to be censorship. Because the university that we accept and propagate in people in every way, it is making more or less profit. The first thing is the original content. It is of X rupees. The forwarded content is X into 5. And the forwarded pictorial content is X into 100. If we are going to do this, I feel that when the money comes out of a person's pocket to forward, which is treating diabetes in free, which is taking it to Baba in free, which is telling us about the scheme in free, which is changing the history of India, I think that is when you know the value of that. Let's be real. Why not? Let's make it a voice. I think they have valuation which is in billions of dollars. We are not going to do that. Dr. Lal, we are doing what he wants. He wants more subscribers. And more subscribers can be found in India. What Sumit is saying, Sumit is at his level as a news platform. We are at our level as a news platform.  He is at his level as a news platform. We are trying to fight this. There is a lot of news channel of this country which is an IFC certified fact checker. As a news channel. And we check 500 stories in a month. Usually. If there is a lot of pressure, it may be 600 or even 400. We check 500 stories in a month, in which we try to tell that the WhatsApp you got, the Facebook you shared, or the message you are sending, this is a fake story. This is the truth. The problem is that now you have to combine and create a model where he only does fact checking. And in the name of fact checking, he doesn't become a sharpshooter, as if something has already stopped. That he will do fact checking on one side and after that he will start sharp shooting. So you need to apply a layer there too. We do it at our level but there is a limit to it. How to reduce it, how to stop it. You need to worry a lot about it. One thing, I live in India for half a year I come to India, I watch TV. When I watch news channels, I have a strange feeling. There are some channels that have become pure entertainment in the matter of news. In them, there is only Shor Shrava, you can call it that some channels are misbehaving. Some people will lie in their studio and you don't agree with them. If you don't agree with them, then you can talk to them in any way. This is permissible. Anil ji, you have had a lot of experience in this business. You should see the Zee business for this. I think there will be only one channel that doesn't debate any political issues. And we don't have this situation or problem here. But the point is that if you take a line in your editorial content and then you go on that line, then you can see the clear views. Now, the people who like it will say that this channel is the best. And those who don't like it will think that this is their propaganda channel. This is easy for us. As it is a business channel, simple things. We cover the issues of business stocks and consumers. If we are biased for consumers and raise their voices, if we are recommending a stock and if our viewers are making money then we get it. But how do I recommend a stock there? Or if I am giving my news content then how do I say it? I have a viewpoint. I can say it in a firm way. Secondly, what is my aggression? Where my aggression comes and I want you to listen to it, that is the problem. Secondly, look at this stock. Fundamentally it is very good. This is the news. These are the facts. But secondly, if you don't buy it, then your world will be destroyed.  This is the difference. So this is the difference. I have to edit it. There is one thing. Emotion. As a news anchor, all of us are human. Emotion should not come and should not come. I will give you two examples. When John F. Kennedy was assassinated, Walter Cronkite, who was a famous tuner on TV, and he gets news that this happened. And he showed emotion. And that was appreciated. When Neil Armstrong landed on the moon, you had emotion of a different type. India wins in a match. Of course, we like it. But every day to bring that emotion, every debate to bring that emotion, and to bring such extreme emotion, bringing emotion and bringing emotion is a big difference. Yes. So news anchor can dance in the studio. I am a human being. So today... I am happy if I get emotion. If my market life is at a high, if my sensex is at a high, and if I jump, I get emotion I am not bringing it. Where you try to bring it, there is no problem. You act there. I am a human. Your question is taking a little bit in a different direction. The emotional part of the discussion, this question you can level any allegation against anyone here on any news platform or maybe for right reasons and those allegations would be true but you should think that if everything is so bad then why does the news channel get advertising? There are some very you know powerful people across the world who are all big advertisers if everything is so wrong and I have asked this question to myself as well if we are all hatred peddlers then if we are hatred peddlers then why do such a big corporate world, such big media agencies, why do they do that? I have asked them many times the truth is that nobody gives a damn right and every Thursday you get a TRP and I am sitting with Mr. Anurag because I am upset that this kind of platform you should use it maybe to figure out this beast called TRP.  This is a corona debate and it should definitely be a panel. So therefore, why is all this argument that there is objectivity, subjectivity, there is misguidance, what is it? What is it that is supposedly being sold? And is it supposedly being sold? Is it being bought? That is why it is going on. This is the same thing that you will remember in Bollywood in the 95th to 90th century when Shreedevi and Jeetendra Ki came into some films. It was a big hit. So every film's plot was the same. There was a problem with the news channels and the plot was not moving forward. I think this is also a period of disruption. People spoke about WhatsApp University. I beg to disagree. Who are the people who are sharing things on WhatsApp? They are eventually our customers, our audience. All this while before the event of social media, journalists were the agenda setters. I will decide what is the headline today. And the difference between the editorial and the story was never there.  Right? What will you show as an eight column story and what will be buried in the 10th page is also an editorial decision. What is written on the opinion page is not the only thing that is an editorial. So when your viewer is setting the agenda, then journalists feel a sense of discomfort. And how bullying is a big deal. It touches people's lives. But a traditional journalist will tell you. So WhatsApp University, instead of denigrating it, I'm not saying pedal fake news or pedal things without fact checking them. But we have to understand that this is what our audience wants. Maybe it's time for us to reorient ourselves and figure out what real news is for people. We must fact check, but we must also present what our viewer wants to see. That's a very deep question. What is real news? But before we go to that, Rahul ji, your view will be... I would like to go a little back and talk. See, after the print, when the private TV started, so we used to go to the field and do the reporting. So what was our hope on us? Our hope was that we were showing everything wrong due to very sharp news. This was our hope for the print. After that, now it's social media. Now we need to talk to the social media on TV. So there's a phase in which the TV is changing. First, there was a phase when the pure news had started. Then there was a phase when we started showing ghosts and spirits. Today, the phase has gone a little to the side. That we are sitting on the prime time. The cheerleaders who have become X party or Y party. This is a little bigger problem. Because a little earlier than this, from a very high point where the private channels were very small, and the activities there, whether it was the blackmailing, or the activities of talking to someone, they caused a lot of damage to the credibility of the TV channels or the journalists. And what we are doing today is probably causing more damage. The people's trust has arisen a lot on us. And they don't believe that we are independent journalists, and we show very clear, pure news. And this reflection is also that when you are showing a video or some news, I think it would be better to give you an example, that the Rajya Sabha TV, because there is a channel about which it is said that it is a government channel, but it is not like that. It is under the Parliament of India, and there is no need to give any special treatment to the government. There on YouTube, we are really good at YouTube. More than 40 million of our subscribers are watching us. When we did a program on the phase the last few days, what were the comments like? The comments came, now this is RSS. These people have completely become Congress and left channels. Where are these comments coming from? From the viewers' point of view. And these people have become against the government, and how can they do this and how can they run this. What are the comments like? Then we will go ahead and create a video on Gold Worker. When we do that program, it is said that this channel has become the RSS of RSTV. Now what are these BJP's? So these comments, I am not saying that these are wrong comments, I am saying that these people's reflection on you is based on their beliefs. Their belief is broken, it is their reflection. And this is the need to think about the beliefs, how much it is necessary to handle them. Sir, I have a question. Please tell me. I don't think it is wrong. You and I can talk about a generation, when we were watching TV, you people were writing on TV in India. Today, when we are sitting here and discussing that there will be a cheerleader at 9 PM in the night, Sir, is it because we are talking about cheerleaders, that he is getting caught or that he is telling? Earlier, the people who were doing it, they were not telling, suddenly one day there was an extinct operation, we found out that the minister is deciding in the newsroom. Then we came to know that the news that you were getting was not news, it was something else and you were going to eat it. So who is responsible for that? Who should we consider responsible for that? And if that is a legacy, then where will things go ahead? Or is there a phase to consider it? There are many phases, this phase will also pass out as you said. Okay, you said that your main question is what examples have we set for the next generation? When I came to the newsroom on TV, the example was that when you go to a press conference, you will not get the gift you get there. Our motto was that I will say one thing in the press conference, that I used to be in star news, I was an input head. And we were doing a news in Mumbai, in which I was telling a news about a politician, I was telling him that this news is true. And he was telling me again and again, confirm it, confirm it more, confirm it more. So the position where I was, he was telling me what is the religion of the editor. And I was telling him as a reporter, what is the religion of the reporter. When these things break, then these things happen today. When we consider the journalist, these are my views, maybe not yours or someone else's, that we are opinion makers, when we start giving opinions, then the other form of the journalist shows us. If we give information and think that Janata is so sensible that she will make her own opinion, then maybe we are more important to the journalism. I would like to ask one thing. Actually, the journalist is a mirror that we all want to show to others. We don't want to see ourselves in it. That's the biggest problem. We have to do the face of this mirror, and what Rohit is asking us, I said when we were outside drinking coffee, you asked me what is the next 5 years. I repeat, I just said that it has been spoiled, it has been spoiled. For the next generation, we have to try that it is not spoiled. Wherever it is, it will stay there. I don't know whether we will improve or not. I will tell you a very good thing. You said that Janata is also sensible. I am seeing a new thing. The protests that we show, I was surprised, the other day when I realized that I was speaking to a reporter of mine, I suddenly saw that he didn't have a mic ID. He didn't have a mic ID on the mic, so I was surprised. I asked him, why aren't you using the mic? He said, if he doesn't do it, then you do it, then it leads to a certain commotion. I felt so bad. I was on the tweet that I had never seen this in my career. The mic that we use, our reputation, our proud image, if you use it as an ID, my face is my ID. I will use it as an ID. So if we don't do it, Janata will do it. But one thing I have seen, extreme ideologies or extreme hate speech are mainstream through media. We cannot do something like self-censorship or like other kind of networks. That something is wrong. What do we need to show it 25 times? I will answer that as well. There is a very straight-forward law in this country. There is law for everything. There is also the broadcasting of the industry, there is broadcasting, right? There are our watchdogs. In one or two cases, one big channel or a big anchor has passed a stricture. Channels have refused to agree with it, So what happened is that every problem can't be solved by law, we will have to bring that to ourselves and until the real crux of the problem is TRP mechanism, this is a flawed mechanism. As long as this flawed mechanism and stakeholders who are benefiting from it, they don't want to get involved because business goes on like this. As long as this mechanism won't be solved, we will go in this direction. Good thing is that you are saying that you can fix TRP's mechanism, at least you are not saying that you stop watching TV. There is no TRP in Jindeshwam, there is a subscription model. Tell me one OTT platform that is making money. None of them are. There is no subscription model in this country, people don't pay for news. You show crime. So the answer I gave them is still relevant. If we show crime for TRP, for our viewers, for which we get ad revenue, maybe 50% is wrong, we show crime, anything. But the 30-40% is right, it is very important. Because if I didn't show that 50% wrong or wrong so-called, then revenue will not come, channel will be closed. And after that I will show right or wrong. So we should also think about what are the helplessness of TV. And it is not that some news channels are wrong. If this was the case, you wouldn't be watching Dine and Nagin, which is TRP. So many people are sitting here in this hall, ask them, what will they buy if they start getting the amount of news? Because they get 1.5 rupees, that's why you buy. The day you get 25 rupees, half the circulation will end. Similarly, if you have to buy the channel that runs on expenses, then half of the people will stop watching and we will come to everyone's home. So the subscription model should go that way. But it is not going. Nobody is ready to pay, but they want to tell us journalism, which we should be doing. But fact-checking and news-gathering is costly.  In the olden days, the environment of my country was not like that.  So it's an expensive business to be in. And I have no hesitation in saying this. It's business, it's not charity. People expect the results of a charity, but it happens to be a business platform. Until you see the business platform as a business, you will still be in it. And most of the media stocks are listed on exchanges. So every quarter we ask questions, how did you perform? And if it's not a performance, then we will grill. Why didn't you perform? So that is there. But one point that is most clear is that all the channels in our opinion, even the viewers are divided. So even the viewers have become like this today, that they want news, they don't want news. People who buy news are not saved. There are very few saved. News is a very boring channel, it only gives news. So where you want news, then you don't want news. No sir, it's news. But this is one of my points. As you heard Bhupin, when he started talking, he said that my show's name is Viewpoint. Because people tell him that we are watching news. You tell us your Viewpoint, Bhupin. We have been watching you for so many years. Now we want to know your Viewpoint. The reason is that Sumit Awasthik starts his show on a social media. Sumit finds out his Viewpoint. Even though he doesn't talk on TV in that way. He is objective on TV. There are three parties, so he trashes all three. But when he is asked on social media, that we are here to know your Viewpoint, then he tells us his Viewpoint. This is my platform, I am telling you. I want to add one thing. I am sorry. We think that they want to know our Viewpoint. Or the experts you have invited, they want to know their Viewpoint. The problem is here. The problem is that I as a journalist, do the viewers want to know Meera's Viewpoint? Or the experts I have invited, what do they think? Because they are expert in the domain. I want to know them. I want to give them an answer. There was a small survey in News 18, in which I was in the channel earlier. This question was asked in our group discussions, in different B towns. Do you want to know the Viewpoint of Anchor, or the Viewpoint of the guests? Most of the time, most of the time, they want to know the Viewpoint of Anchor. And the public answered that, I listen to XYZ Prabhakta somewhere else. But I can't hear your voice. What is your Viewpoint? This is the part of the study. There is a Viewpoint of Anchor in Prabhakta's elections. There is a Viewpoint of the channel. This is the survey of the guests. This is the survey. This is the survey like a bark. So... So what? Our currency is the same as today's.   I used to say the same. The way politics is going, last year and the year before it was said, that why is there the shift to the right? The world over.  That can be said of the media too. There became a cozy club. That club is being challenged. Correct. And so this discussion is happening. And I think it's in a positive light. Yes. We were talking about this. Since the time is over, I want to make sure that let us not be completely cynical and dismissive. No, absolutely not. Despite all these challenges which are there, I am very happy. I am very happy. There are platforms. Like what Mr.Millan is doing, my compliments to him. He has started a great channel. The data that I am talking about, which I also want to challenge, I don't have any mechanism. People are doing it according to their own needs. At least they are getting screwed. This is getting screwed at some point. I believe that at some point you have made so much pressure on the international media. You are also in that direction. One question from you. You have been asking a lot of questions from us. Time is running out. I can't leave my anchoring. When I am returning to the topic that I am talking about, Dr.Lal, because we are all anchors. We all give our opinions. I don't know about your opinion. If Prime Minister in this country starts a program after 70 years of freedom in Swachh Bharat, is it a propaganda or an editorial content to show that? Please. I think it is absolutely the duty of every citizen to participate in anything, the ruling party or the government that has been elected. And TV networks are also part of the country. So the guidelines being set by the country. There is nothing wrong in it. It is not propaganda from that perspective. You may have strong opinions about the person who is running the country or not. That is separate. Quick question. Last one. Then we will open it for the audience. I have observed a lot of celebrities who are primarily celebrities in the movie space or some other space. They have overnight become experts on almost everything from intellectual intelligence to zoology. And whenever there is an issue with the TV networks have, they go to some celebrity, some filmmaker and he has a well thought out of opinion which is totally wrong, but he is imposed on the public. Is this justified? I think we are a country that has a lot of conversation and debate on chai stalls. Everybody in India is an expert. You can ask the person who is making your tea, who can ask your maid, she will have an opinion on who should you vote for and who you shouldn't. I used to live in a society of journalists. Somebody came to deliver pizza. The lady asked the pizza delivery boy, who will you vote for? He answered and someone said there was a full discussion on why a party should be voted. Celebrities are faces, that is why they sell. Everyone in this country has an opinion. Same thing. We are living in a time where big journalists of the country happily tweet in the evening, I was in a taxi and I spoke to a driver and the driver told me that he was going. And he decides that now I want to take the opinion of the country in this direction. Same thing. So when everyone can give their opinion, then everyone's opinion is taken. Then what is the fault of a film star? If people want to see Sumit Ji's show or Ruhi Ji's show, they want their opinion. Similarly, they want actors too. The public face, the public figure, they enjoy listening to their opinion. If you ask, it is possible that we have to ask 50 actors, let's make it 40, let's make it 45. We will give them 5 opinions, we will give them their opinion. I have so much to say, we will show the opinion of the actors which is of no use in your opinion. Don't listen to their opinion. We are living in democracy, reject them. Our problem is that we all give our opinions. We don't know how to say no.


My little buttercup. Huh? Unlike CDs, DVDs have digital locks, and the very act of breaking them, even for a legitimate purpose, can be a violation of the law and get you sued. But every three years the Copyright Office looks for problems with the DMCA and grants exemptions. This year, public knowledge is asking the Copyright Office to grant an exemption, allowing people to break digital locks on DVDs they own in order to copy them and wash them on devices they also own. We need your help to convince the Copyright Office that people really do have a legitimate desire to watch movies they own on DVD on all of their devices. Click here to tell the Copyright Office that you own movies on DVD, you own devices with screens, and that it is stupid that transferring the movie from one to the other could violate the DMCA. That way... My little buttercup.


Hello everyone, this is Amit Dhan and on this channel you'll find financial travel and more videos. If you like what you see, please do the thumbs up and consider subscribing. Thank you. Hello everyone, if you're watching this channel for the first time, you're watching one of my travel videos. My name is Amit Dhan and you're watching one of my Asia trip videos and I'm in India right now and today I'm going to show you a few ministry headquarters in New Delhi, India. So I'm going to just drive through them and then we'll have a glimpse of the Presidential Palace in India. So we'll just drive around. However, I won't be able to have a closer look or get inside the Presidential Palace. So let's enjoy the video. Thank you. So the building you're looking at right now on the right is the Home Ministry of India and the building I'm passing right now on the right side is the Finance Ministry of India. The building right now you're looking at is the Defence Ministry of India. Looking at right now is the Office of the Minister of External Affairs. The building you're looking at right in front is the President's House of India or the Presidential Palace and this palace actually has 340 plus rooms but we won't be able to get closer than this. This is as close as we can get so we're just taking a different direction from here. For the next one and a half minute I'll just show you some scenes from this area and if you like what you see please give it a thumbs up and subscribe. There will be more videos coming up from my India tour so I'll see you shortly. Thank you for watching.


OK. Sorry for the delay. We were setting up some Windows environment. So this is how I'm going to present. So I hope you all can see this screen. OK. My topic today is building your Spark Merchant Learning job online like Lego. So what does this mean? This site, we call it Spark Merchant Learning Workflow Visualization. So this sounds most folk, but as I explain more and more, you're going to learn how each of the words means. OK. Before I speak, I want to introduce a bit about myself and my company. My name is Hu Dawei. Or you can call me David. And I work for Qbida. I'm a system engineer. So Qbida is an enterprise analytics company. So we provide one-stop enterprise analytics service. It's based on open source. And all these Hadoop Hive Spark we are the mainstream technology we've been using them. And also it's data source and platform agnostic so that you can basically connect any source of data and output it to anywhere. And these are our main features. So we connect your data to the Hadoop to leverage the power of a Hadoop. So from there, we start to discover your data, transform it, and figure out valuable information out of it. Then at the end, we output the analyzed report so that you can have this nice intuitive web interface to know what your data really is about. All right, this is the intro. So let's jump into the topic here. Spark Merchant Learning Workflow Visualization. So what does this mean? So it's basically you can create and modify and execute a Spark job with just a single drag and drop feature through a website. And all happens immediately. So I will give you a quick demo first then you know how it looks and I will explain the details and the technology we've been using to support such a model. OK, so I will show you our cluster. So as you can see here, this is a workflow. So you can see it's all being linked, connected together with this directed async graph. And all the work from the reading a data frame to eventually apply different machine learning algorithm. And get your result, get what you want. And this one I have already ready. So you can change it like this so that all these parameters you can change and also for those things. And you can even see the preview of intermediate data sets so that after each step you will know what happens to your data and does your purpose to build the workflow is correct or not. All right, this is going to take a while because it's running actual Spark job. So I will go back to the slides. So this is a structure, how the structure looks like. As you can see, we read the data frame first, then filter some columns by input some conditions. The conditions meaning you can just put a SQL script. So Spark transformation, which accept a raw SQL script. And all the results are having this Spark. We wrap all this Spark API into this nice intuitive interface. OK, so here we have a couple of examples of how you can customize your nodes. Take this for example, which is SQL transformation. So it requires two parameters. One is called data frame alias. So in here you input one. So this one will create a time view in the Spark cluster. So this one following the second parameter will be able to reference this one from here so that Spark will know how this data frame is being connected. And here's another example, the split. The split is to a so that we have a data frame. You can split into different two set of parts that based on those parameters say you want to be random or ratio or seed. The result preview, as I introduced before, it is for the intermediate result. And of course, also you can show the final results. Maybe I can show you about this part. You can see here we have many other nodes that come in from here, those are called categories so that you can combine different kind of nodes from different categories to do what you want.  So here is that you can just drag and drop and without running a single line of code. How does it work? So as you see previously, this structure, when you run it, it immediately runs. And even though it's running from the web, it does actually launch the actual Spark job on the back and on the cluster. And secondly, each of the nodes is being parsed into an exposed Scala code snippet on the fly so that each node represents a part of the Scala code. So once you change the parameters, and the Scala code is going to be changed. And the thirdly, the code is sent to cluster by HTTP post request. So now you might be wondering how this is possible, right? Yeah. So here is if you understand how Spark works, you will have this naturally, you would have these doubts that this thing might not be impossible. Because why? Currently, there are two ways for you to run a Spark job. One is through the Spark shell, another one from a Spark submit. So a Spark shell allows you to open a shell. So in there, you start to write your code Scala or Python or R, right? So I believe I don't know how many of you have been using Spark with your hands. OK, not so many. I guess I need to introduce about the Spark. So Spark is a computing engine that based on the Hadoop. The Hadoop is a distributed file system so where you store your data from different servers. It's different from the retinal, what's the word again? So like Oracle or Postgres. So those are structured data. But normally, your data is not so structured as per se, like CSV or JSON or even image, some kind of stuff. And they are not that important. But there are values out of it, but they are not that important. So you don't want to store them in a database that's so expensive and so straight. So the file system is a natural place for you to place those stuff. So HTVS Hadoop is based on this idea. So now we come to Spark. Spark is a computing engine on Hadoop. So which means that it can gather all the data, read the data from Hadoop and doing all the transformation and filtering and other stuff so that just like how you do to a database, the data in the database. And that's the basic background about Hadoop and Spark. So now two ways to do a Spark job. One is Spark Shell and another one is Spark Summit. So Spark Summit is that you need to write a Spark application first, I mean a truly complete Scala app and you need to pack it to be a jar. Then you sum it with the Spark jar, the Spark Summit. So Spark will do all the like uploading and figure out the dependencies and et cetera, et cetera. So now here comes the limitation that all those two things require you to programming manually. It makes sense, right? So if you want to go to run a to using Spark to analyze some data, you need to get your code first and go to Spark Shell to paste there or just manually typing or you build it as an app and submit from the cluster. And this way, the second one part is going to require the cluster access, which is actually quite limiting because not all the people are administrator. So they are not familiar with the server environment or something like that. So this actually very limiting the power list of the Spark. So it makes you think that is there a third alternative way to that get a full use of the Spark and also make the access to it. It's being easy and accessible from anywhere. Now you might be thinking the third way. Naturally, if you've been a developer, you know most of the request communication by the HTTP request, right? So here, if only we could just use the good old HTTP request that is easy to use. So standard for heavy devices, you can use your phone, your browser, or whatever tablets. And it's easy to access. So you can use any kind of tools like Kerr, even a software like Postman. And it's lightweighted. Whenever you want to make a pull request, you don't really need to add any dependency to it. So because most of the language and tools are already having this supporting the networking, right? So this is the good things about HTTP requests, the web. So if you think about it, though, is there what if we can combine Spark with the web interface? And that's what Levy does. So introducing Levy. Levy is a patch project which is still ongoing. But it has already really nice rich features that you can use. So it is a web service that enables easy interaction with Spark cluster over a web interface. What does this mean? This means that by the API provided by the Levy, you can use different ways to submit your Spark job to the cluster. One is to pre-compile jars, which you already have, that are good. OK, we are away. So we introduced this snippet code, and certainly the Java Scala client API that where you can write your Scala job embedded within the app and using the API to submit it. And all this stuff you can find on their website is very detailed and very well-organized. So you can read about it later if you are interested. The second feature is that the Levy is ensure security while the secure authentication communication so that you don't need to worry about your data being linked because that it's sending through the internet and it's easy to integrate with a lot of authentication libraries on the Hadoop, like Ranger, something like that. Or you can catalyze with your service. And certainly the most important one is sending most of the APIs as asynchronous requests. What does this mean? This means that once you issue an HTTP request to the cluster, it returns the response immediately with a body to indicate the current status of the cluster. So it will not wait for your job to be finished and it's going to give you a response. So if you think about it, it's quite reasonable. Most of the jobs are actually running from minutes to hours, which is very time consuming. So this one, by doing this, you can immediately get the response and figure out what's the next step to do based on the response. All right. And here is the Levy infrastructure. As you can see, here is the HTTP interface. And here is the client, which you need to embed it, integrate it with your application. And sitting between the client and the cluster manager, which can be most of it is yarn. And we have a REST server. So whenever a request comes in, and the request server is going to sign this code or jar directly into the cluster manager and starting the new job. And it's going to keep getting the latest response. So keep getting the latest status from the cluster that you can use. That's how you get the information out of it. So now we know the infrastructure of the Levy. How do we integrate with our current design, which is the machine learning workflow? What we did, if you want to integrate with Levy, what you need is to enable the Levy service in your cluster. What you need to do is to download the Levy and set it up. And once you set it up, you will have a Levy URL and port where you can communicate with Levy. After that, you can send the code snippet to Levy REST server by the post request. And as we talked before, it's asynchronous. So we will do the post status for the latest executing session or the statement. OK. Now the next step, we'll check about how the API looks like. It's actually quite simple. There are a lot of APIs. But those two are the most important ones. That if you want to use Levy, first one is to create a session. It's very simple that you post slash sessions to the Levy endpoint and passing a body with a kind parameter. And by default, it will be Spark. If you want to use Spark, you can ignore this one. But if you say, I want to start the session in Python or R, then you need to specifically this to be PySpark or Spark R. All right. Now once you post it, it would immediately give you a response in the body that will have an ID of the session. So here, you can use in the second API. This one, if you call this, it will give you the status of the current running session. So at the beginning, if I create the session, it will be busy starting. Once it's been created, it's going to show you it's been created. Now you can do the following things. OK, the second one, which our platform is tightly relied on, which is creating a statement. Similar like this, it has to post a statement. And in the body, it only requires a code. And you're passing the code you want to run. It's plain simple. It's just a string. A string that represents a Scala code. Or if you prefer Python, you can write Python code. But I'm familiar. I'm good at with Scala, so I just choose Scala. So now, similar like the first one, if you post, it will return a statement ID. So you can use in the same RESTful API. You get the statement of the last sending code. Yeah, so giving those information that how the review works and how our workflow works, then you might have a clue at how those things being combined together. So let's review this whole picture here. So this workflow is combined with different type of nodes. You can think of the node to be a configuration that eventually going to create a piece of Scala code snippet. But how? From a visualized node to the actual Scala code. And here is if we review the previous one, we can figure out the relationship between all the structures. And from there, we can start to design our data structure. So as you can see earlier, it workflow consists of a sorted node sequence and metadata. And metadata is like a description, the ID, or name that you just for the other information. You don't really care about if they don't really matter when you run the Scala code. The node has a certain input and output with different types. So go back to here. You can see. Let's take a feature column as an example. It has an input. It doesn't say, but the input is a data frame. So we use different colors to indicate different type of data frame. So data frame is a you can think of a abstracted table, like loading to memory that gather the data coming from Hadoop. So you can think of the data frame is the main abstraction of how Spark would transformation and actions. All right. So it accept an input. So a input from a reading data frame. And it output two things. One is also a data frame that I want a sub data frame. I do some filtering. Then I output the result. Makes sense, right? And it also outputting a transformer. What is a transformer? It's a reusable structure that you can apply to the other data frame. So let's say this configuration, this filter column is so powerful, so useful, I want to reuse everywhere. But I don't want to create a node for all over the place. I don't want to duplicate that. So that if you don't want to do that, now you can connect it to a different data frame than do a transform like this one. So that that's the way how you can reuse it. Enough of the details. And this is a workflow. And they are the nodes. And with these nodes, sorry, one more thing. We are talking about two type of output here. Data frame and transformer. But for machine learning, you also have trained model or like the feed one. The feed one, once you feed is data set, it produces a model. Or some other, like this one is a metric, which is just a single value that indicates how good your model is. All right? The child nodes need to know the parent node output type and the position. What does this mean? Go back to the example here. You can see the split. It also accept the data frame from here and split the data frame into two. Now we have a node that produce two data frame. So that let's say a child node so that not only need to know the parent nodes type, which is data frame, but also need to know the position of that so that they don't get confused. So by this one, you know that I need an ID is this and a type is data frame. And output ID is two. Let's say it's two because one, two here. That in this way, you can identify the exact parent of the child node is. OK. Each node need a code generator. What does this mean? As you can see earlier, each code has a ton of, I mean, not ton of, but how do you put this? Because based on the purpose of each node that they do all sort of things, right? You do transformation, do machine learning, and apply different kind of algorithm. And also, you want to output the data into different places like Hive, HTVS, or S3. So all sort of different purpose. So naturally, they would require different parameters. So how do we combine the code and parameters together to generate a valid Scala code? And here is our data structure design. Before I introduce, anyone here know about Scala? OK, not so many. All right, so I think it's still OK to indicate how it works. A case class is a kind of a class that only contains data. You can think of it that way. So a data structure, a struct like C or whatever other things is. If you know about Java, don't have this. But Kotlin has a data class that was whatever you know what it is, right? So case class, we have a workflow. It's ID, name, and list of nodes. So you can see the type here is Victor. Victor in Scala is an immutable array. You can think of that. It's immutable so that once it's been created, you can never change it. And the list of nodes, and each of the nodes have this structure. So you have other ID, which indicate which one first, and which one the second. It has a name and a category so that you can see one more thing here. So you can see we have different categories and having different names. So the output would be the category, and the name would be the actual node name. But in this way, we can locate a node that precisely in our code. OK, here. Code-generated design. In our case, we call it statement similar like the one you see in the Levi API, create statement. But in this way, ignore the name. You can think of a builder that taking a node as a parameter, then output a piece of string, which is a value Scala code. And you can see, because we have all different nodes, so we want to have a common interface to ensure that all the nodes having this similar behavior say that they all need to be having this snippet interface that as long as this node implement this statement, they have to implement this snippet so that it will give out a code string. And here, we have parallel output types, self-output types, and prime keys, and the snippet. Those three things are the first three things are actually like serve as a validation role, because we need to read the data from the database and convert it into the previous one, sorry, into this kind of structure so that in the statement in the builder, we want to validate as the data is valid or not. Why? Because when we collect all the configurations into the DB, we've been using MongoDB. We cannot really guarantee that if the data input by the user are valid or not, so we do a double check here. So the last one is one that matters, and this one produce the actual code. All right, before you start the structure, sorry, yeah? I have sectors of strings, not sectors of all types of all Spark types. You mean the output wider? You mean here? I call it Spark types. Actually, that's a very good catch. Right before I do this presentation, I do this snapshot. I notice that string is a bit flexible. Is that what you're concerning? Yeah, so type as we mentioned earlier. Yeah, so a better refactoring can be we use object that object is quite like a class, but a single thing class, and it's very, it's type guaranteed. So we can rely on the strong type system in Scala, not by this string. String is too flexible, even though we do check if it is comparing to a data frame or other stuff. But yeah, that's a very good catch. Thank you. Here. Here? Best design when you need to instance a different statement. Those different statements will be, you will have a class with a huge statement for different kinds of things. It's better to have a trade class that is seen. So you implement the statement everyone inside the same class file, and you prevent the problem when you try to do a lot of matching, because you can use the standard match. OK, I see. So your question is, how do we, based on the kind, based on the type, category, and the name, how do you figure out which builder to use? Is that your question? Or you are? For the statement class, why is it not free? Why is it not, sorry? This statement class that you write can be converted to a trade. We are there in the node mode. I'm sorry. I'm not quite getting the question here. Abstract class, if you do a still trade, it's better also in order than the other code, because you must. You mean why it's abstract or? Why is not free? Why are we using a type safe approach to do these kinds of, the statement, you are using the statement as the base for providing the code that will be sent to the code. So the most important part for you is to have that the statement is much more possible than type safe. One problem was the key parameter. You can use, for example, Enumeratu that Yeah. A better approach for the keys. Right. So we're using a seller trade statement. So you will define all the statement inside the same class file, Scala file. OK, actually, I'm only showing part of the statement part. I also have a company object, but the screen is too small. I think that's not irrelevant. Maybe I finish this thing. And actually, I'm not quite sure what your question is. So maybe I finish mine. And we can discuss more about how this design approach. So here is the abstract class statement, which is a builder so that for different kind of nodes, you will have different kind of builder to inherit this one. And in terms of, OK, let me show you the code first. So previously, you see the code structure. Now you are wondering, once I get a node from the database, how do I find the right code generator or builder? One way to do that is because we have in this name and category, one way to do that is to have a big match case case. And this is going to keep continue growing as long as you have more nodes being added into or by something else. Here you can see we are using a reflection, which is a Java concept that you dynamically find in the class during the runtime. So how do we do that? So you can see here, based on the category and name, we find that this defines a class path for that builder. And here we have this concrete statement. Now the following would be find that statement and the class constructor. And eventually, I create a new instance of that statement. You can see here, I pass this, which itself is a node. And you can also check out, if you remember, the abstract class statement as a second node. So in the node, I pass myself into the statement I just found out based on the information, name, and category. We found out. Now we have a statement instance. And the statement instance would call snippet, and it immediately returns as it calls snippet. Make sense? All right. OK, here maybe this is an example of how this co-generating from raw data from database into a real code value scalar code. Here you can see, here I have a node, which is a node number three. Order ID 3, you can see. And it have category transformation custom. And we're going to convert this underscore into a dot so that match the Java mechanism. No name and description, you can ignore it. Object, just an object. Output, which is an array. So this means that this node cell cycle transformation would output a data frame, as you can see here, which itself is a string. The parent outputs because one node can have one or more or no parents. So that is also a list of objects. So the object structure is ID, parent ID, which is order ID of the parents, actually. And output ID indicating the order of the parent output, like the one. Remember the example I told you, this two same type of the data frame. So you need to figure out which one is the correct one so that output type is data frame. So this is the raw data stuff in the Mongo. We've been using Mongo because it's quite convenient to do this kind of stuff. Here is a SQL transformation just for that node. As you can see here, we have all those three because inherit the statement. Now we need to override those methods to make this node work. The cool thing about Scala is that when you define a method in the abstract class or a treat, in the subclass, you can actually read not using the method, but using Vow. So that will save you from repeating complication that whenever I need this, I don't need to recompute it, but directly compute it once and cache it. Next time, I directly add the value. So here's a snippet, the most important one that here. This is a template. So based on all those parameters, I have got this parent data frame and self-dataprint output and produce a code that which is valid Scala code. So to answer your question, the stream part, right? Here you can see. Yes? Of course. Sorry? Type. Yes. Yes? Yeah? OK. So yes, Scala has a, you can define the function type. Yeah, of course. OK. That's the reason I designed like this because in Scala, you can go really deep and using all these nice features. But for me, I think this is the most understandable way to implement this one. I think it's, forget about this Scala code, but even if you are a Java developer or a Python or Ruby, I think this, you can implement this same idea into other places. I think not because you can define a function in Scala as a type, you should. I don't think you should use that to just prove or just showing how good this code is. But I think for me, the readability is the most important thing. Yeah, so that's my approach. That's why I choose this approach over that one. OK, so here is all of these are big technicals. So if you have questions, we can discuss later. So a simple one, a previous one is quite complex. And the output of the final Scala application can go to hundreds of codes. So I create a small one that what it does is quite simple. Read the data frame, do some transformation, and write into a high table. Right now, the write data frame before goes to a high table. So here is the output of the Scala code. So as you can see, if I piece all of this code directly into the SparkShall, it would run perfectly. Because the Levy maintains this thing is you can think of like online version of SparkShall. Online version of SparkShall. So you can see the name I define. So this is the first node, second node, and third node. By giving a standard format of the variable, I can reference the parent nodes easily. So all the data frame outputs having the same structures that start with output and with the other ID, indicate the nodes, and the type, and the output. So in the second one, as long as I know the parent type, and ID, and the position, I will immediately calculate how the parent node can be found. So here I have been reading the data frame, select all from these transactions. And I want to find all the CDA, and I write it into CDA transactions. That's simple. OK. This project is not all the ideas that come in from us. So this one, the C, there was an open source project called Seahorse, which does similar things like what we did. But we've been being in parallel, so we don't know each other. But once we found out, we are very excited. And we go to their website and check out their repo. And the same situations. What they did is actually go really, really deep. That together with the JVM and the Hadoop platform, and they redefined a lot of things. But for us, and you can, if you go to the project, it's actually lasting almost two years and hundreds of man powers to developing these things. For us, I want the same feature. And by studying the JVM, I figured out we can do it in a way simpler ways, which is we manually hard coding every template of the operation so that we can put them together as you want. So if you want to know how the full feature, you can go here and check it out. What we did is similar, but the way is there is only cost like three months. The back end is all on me. So yeah, the design and the back end implementation, that took about two months. So comparing one month, two months to a full two years, I think that's quite an efficient way by using DV. OK, take place. What do we learn? Do we learn from this project? First of all, the very impressive one is Levy. I think it's really opened a lot of possibilities to how you build a Scala job. So it really unleashed the power from previously. You can only log into the server and write your code and submit it. And in this way, Levy is much more flexible and more access. Second one, during our implementation, because we need to wrap heavy like most of the operations using Spark APIs. So we found out most of your needs actually it's already been considered in the Spark API. So I think the most impressive thing is that they really started well and really performed. The performance is so good that I think every time you want to implement something you own, you want to do some data, think twice to see if Spark has it already. Go there, documentation, list, and have a check. If not, I mean 99% of the chance you already have something there. And thirdly, the immutable data structures, which is not a common concept outside of Scala or outside of functional programming work. But it's very important that you know this because once you force yourself into a think of the data as immutable transformation, instead of getting a data, I want to modify it, then you can see your whole idea about programming can change. That's why you can see here in the code I showed you, I only use the immutable actions like Victor, map, or set, those kind of structures that once you create, you can never change. And all my code are using VAL instead of VAR. I think that really guarantees that in the generated Scala code having very few bugs. Because the second layer of the Scala, you can see here, this one is actually quite small. Some of them can be close to hundreds of lines. Take one, for example, we have a node called handle missing values. It's a common operation that if you do a static, which is that if you have some rules that have no values and you want to do something about it, you can replace with a custom value, or you can replace with the mode of the column or the medium of the column, that kind of thing. So for each different structure, you need to calculate that value to field. And that is to take a lot of effort to build that. This is a quite simple one, but this serves the demo purpose. All right, here are all the references I was talking. I mentioned earlier the Levy Apache project, the DeepSense AI and Seahorse, and our previous talk by our CEO, Britain, if you're interested, you can go to it. All right, that's all. Thank you. Any questions? Which part? Oh, this one. OK, this, yeah, of course. OK, here are lots of methods. That's the statement. Those are the APIs. Those are the methods that each of the child builder need to be implementing. But there are a lot of helpers, and this instance method need to, maybe I can show you right now. Is taking the single string. Yes. So OK, here's a. The side is taking a single string. Yeah, so. And retour the sample. Yeah. So the primer key is just a list of strings, which is a guarantee that I need two parameters keys. One is the time value and expression. And by doing it, and here we have a map get. And this. Yes. Yes, so yeah. This get is accepting a string. And using this string, you find that value from the database. So this get is actually a method defined in the statement. OK, here are some scholar code. I'm not sure you can follow, but I will try to explain. So this get accept a string, and goes to the from it, and get value. That's simple. Return the stuff. OK, sorry. On your picture, you have a graph of operations. Yes. It looks like the. That's a illusion. It looks like you can send them nodes that have no relations. So you think you can accept it, but we actually have an order defined with each other. So in reality, it's still a linear processing, each of the node. And you define it manually. Actually that part, we have two separate APIs. We have this one is a backend service. For all defining the order and collecting the configuration, we have another Node.js app to do all that stuff. So the right, the configuration database and the read from it is actually separate by two different apps. That's all that's the structure we have. So the order is being defined by the front end. I'm not sure if you have. Some logic on front end, or. The logic. Or by user input. The user input. User must put some numbers in each node. No, no, no need. We have this algorithm to calculate which node should be go first. Yeah, so every time you're adding a new node, I'll remove one. So this new order will be regenerated every time so that you always get the correct one. Yes? So if I have to find a pipeline to define, can you short something to make it a pipeline like the one I cross instead of going one by one? Because you have a code generator. You can easily just generate a bigger code. Yes, so right now it's one. I'm not sure if you care because we generate the scale of code. The scale of code I send to Spark. Yeah, but I didn't get it. It's represented in a graph. And the steps in the graph are aggregated by Spark directly. So if you have some easy modifications, such as fields that are all marked, they are executed in the same steps inside Spark. So it's not required to aggregate them. I think each of those blocks, each node, is a separate Spark. Well, it generates a single one. Yes, so as you can see. It's a single one. Yeah, every single one generates some kind of code. And every one generates a big file. It's your Spark job. It's a compiler because it's in the same size as we need to compile the data. Yeah, yeah. So since it's a run-out, so you can see on the B, how do you capture the errors into any event you control? Actually, yes. So that feature is provided already by Levy. So whenever you make this guest status request in the response body, you can by checking some certain keys, say status, log, error message. So if say the log fails, you already know it's something wrong, right? So you go to the log. It having actually a pretty nice structured log information, you can go there and check. Only polling. Yes, but also since it's just endpoint with the UI and port that, so you don't need to use your code to go to make a guide response. You can just use the ending tools to get results. Or even using the browser or some HTTP client, like Curl or Postman, whatever, you can get a log. All right, any more questions? OK, then that's it. All right, thank you.


We're here in James Park and the kids are absolutely amazing. They're very excited. They're so enthusiastic about reading And I'm so thankful and I've encouraged them to go home talk to their moms and dads Make sure they encourage a whole family to read and when they're feeling comfortable if they read to their younger brothers and sisters That's a great way to making sure the literacy is passed on and once you become a strong reader the rest of your life Is gonna be that much easier Oh These are pretty comfy So what happens is you guys come in here The fact is that we're one of the best literacy Communities in the world. First of all, we have amazing teachers Second of all, we have parents at home. We're really encouraging their children to read. It's so important and as well We just as students who are super excited about reading guys over there. Do we like reading? Yeah, they love reading and it's so important that we share that love with Burge Columbia is all over so we're really pleased that reading in Burge Columbia has done so well I'm looking forward to me containing and improving that going forward Have you read a book about the Hobbit before? Some of us have read it before? Oh my I'm impressed. Well done


Three sure-fire ways to fail at losing weight. Like other scientific fields, the science of weight loss is continually evolving. What's true is that no single diet approach, whether low-carb, low-fat, paleo, or vegan, with no or varying amounts of physical activity, will work for every individual. It's also true that within each approach, there are people who will lose weight, as well as people who will gain weight. Even more confounding is the fact that those who successfully lose weight have an extremely high failure rate when it comes to maintaining that weight loss. This dichotomy leaves scientists baffled as the search for the ideal diet and weight loss maintenance strategies continues. Scientists, doctors, and nutritionists all agree, however, on three things that always lead to failure when it comes to weight loss. Unrealistic expectations, the quick-fix approach, and the starvation method. Number one, unrealistic expectations. Unrealistic expectations of ideal weights are often enough to keep people from losing any weight at all. A healthy, sustainable weight is often a higher number on the scale than one would tend to think. Accepting that a 10% weight loss can improve blood pressure and blood sugar and reduce the risk of cardiovascular disease and type 2 diabetes may help overweight people to make small steps towards that goal. Number two, the quick-fix approach. Unfortunately, many view weight loss as a quick-fix as opposed to a long-term solution for improved health. Clearly, a change in mindset is necessary not only for weight loss but for long-term weight maintenance. Number three, the starvation method. Diets that are too calorie-restrictive are destined to fail. In addition to your increased hunger, your body works against you, which makes losing weight even harder. Whatever your diet choice, recognizing that we often eat in response to stress, boredom, or emotions may help to avoid setbacks. If you are serious and determined to lose weight and keep it off, take a slow but steady pace. If you have any questions or would like to speak to a registered nurse about the right weight loss plan for you, please call, visit, or email us, and one of our knowledgeable staff members will be happy to assist you.


Snatch drunk. Ninja Cop is one of the best names of any game I've ever seen, but sadly this game is only referred to as Ninja Cop overseas and in North America it got the lame ass title Ninja 5-0, presumably a play on Hawaii 5-0. This game did not sell worth a damn and that's really too bad because this is definitely a case of a great game that just fell through the cracks. That's right, this is one of those 2D platformers where you're given a ton of abilities. You can slash with a sword, throw fireballs, use a grappling hook to reach places, cling to ledges, upgrade your weapons to even more powerful stuff, fill up the bar up top and unleash a clear screen attack, on and on. Ninja 5-0 was developed by Hudson Soft, who developed a Super Nintendo game very similar to this, Higane, another game where you're overpowered to the gills, but this time around you're allowed to be a bit more deliberate and explore a bit. You search around sizable levels defeating enemies, some of which are badass ninjas in their own right, and you have to find these colored keys to unlock doors so you can move on to the next level. Not only that, but you can't just hack and slash like crazy on some of these enemies. Occasionally you have to rescue hostages without harming them, match the pattern here and take out the enemy carefully. And even without hostages in the way, some of these bad guys are pretty tough.  The grappling hook is a huge part of the game as well, as it's tricky to be able to reach certain areas without taking damage, and that's what helps make this game stand out from other 2D platformers. Take this segment here where you have to use the grappling hook to get up onto this ledge while avoiding enemy fire, then be careful enough to take him out without hitting the hostage. That's what makes Ninja 50 seem like an upgrade over the stuff that influenced it. It's subtle, nuanced stuff like that. But not only that, you get 20 levels of this too. Now that's awesome. And while this game is challenging, it's not to the point where you want to throw the controller. Of course it helps that the game looks and sounds great as well. The story totally doesn't matter because who cares, but it does provide some nice visuals in between the stages. The music also gets the job done by matching the action appropriately. So yeah, Ninja 50 is a great frickin' game because not only is it able to fluidly combine elements from many games before it, it's able to stand out with great level design and even better enemy design. So it's not just another replicated derivative game. But if you're familiar with how I structure these videos, you know what's coming next. Yes, that's right. It's the depressing news that this game goes for an average of $66 on eBay. But still, if you're able to find a way to play Ninja 50 or Ninja Cop, I highly recommend you do so. It's like the next evolutionary step in old school NES action platforming, and I really wish it got a sequel.