Hello and welcome to the session. My name is Mansi and I am going to help you with the following question. The question says three coins are tossed, describe three events which are mutually exclusive and exhaustive. So, let us see the solution. First of all, we see the sample space when three coins are tossed. So, sample space will be H H H, H H T, H T H, H T T, T H H, T H T, T T H and T T T. Now this we can get three heads on three coins or in the first two coins we can have heads, in the third coin we can have a tail and so on. H denotes heads and T denotes tails. First of all, let A be the event of getting no heads. So, the sample space for the event A will be all those elements where we do not have any heads that is just one that is T T T. Let B be the event of getting exactly one head. So, sample space for the event B will be all the elements where we have just one head that will be this and this and this that will be H T T, T H T and T T H. Next, let C be the event of getting at least two heads. So, the sample space for C will be for the event C will be H H H, H H T, H T H, T H H that is this. Now we have to show that they are mutually exclusive. We see that A intersection B, intersection C is equal to phi. That means we have nothing common in these three events. Therefore, they are mutually exclusive. Now we have to show that they are mutually exhaustive. We see that since A union B union C is equal to the sample space therefore they are mutually exhaustive. So, we see that whenever we have events such that their intersection is equal to phi then they are mutually exclusive. Since we see that in three of the events no element is common therefore thus this will be equal to phi and they will be mutually exclusive. We also see that A union B union C that means when we combine these three events we get the main sample space. So, in such conditions we say that they are mutually exhaustive. So, our answer to the question is getting no heads, getting exactly one head and getting at least two heads. So, this is our answer to the question. I hope that you understood the question and enjoyed the session. Have a good day.


In commemoration of a 2023 World Television Day, journalists have been charged to kick out the dissemination of fake news and incorporate fast-checking in their news reportage. The celebration-themed television of Strategic Two for Conflict Management and Nation Building and Sub-Theme Media Space for Information Access in Nigeria was organized by the Consolidated Chapel of the Nigerian Union of Ogun State Council. Shergung Olukoja, Chairman of the Consolidated Chapel, while speaking at a celebration held in Iweroi Abyokutap, said the media will continue to be a strategic tool for shaping the world and bridging the gaps among diverse communities. He noted that the media are responsible for championing development and the society and support the government. Also, the Chairman, Ogun NUJ Council, Comrade Wali Olarawajo, disclosed that the media has the power to inform and inspire the people through television and other media. In her keynote speech, Tong Okewale Shonair, CEO, 91.7 WFM Radio, represented by Esther Alarebe, the program's manager, asserted that the media must continue to be objective, open and responsible in delivering accurate and trustworthy information in spite of the difficulties that confronted. It is a reminder of the new patterns of responsible broadcasting, promoting quality content, and nursing programming for positive social change. Television will continue to be a strategic tool to shaping our world and bridging the gaps among diverse communities, championing the voice of the people, and promoting adequate information and helping policies, programs, and projects. We celebrate the profound impact of media in providing access to new information in a year. Through television and other media platforms, we have the power to connect, inform, and inspire in a country as diverse as Nigeria, because it plays a pivotal role in ensuring that information reaches every corner of our society. We know the vital role that the media plays in building and nation development. With the advancement of technology, the media space has expanded, allowing for greater access to information, which has led to increased awareness, education, and empowerment amongst Nigerians. However, it is important to note that there are challenges, which must ensure that the media remains unbiased, transparent, and accountable in providing accurate information.


And away now from labor martyrs, the former Zamfara state governor, Abdulaziz Yari, has vowed to continue his push for the Senate presidency. Yari, who addressed journalists in Abuja said nobody can stop him from vying for the Senate presidency because it was his constitutional right. He insisted that no individual has the exclusive power to decide who leads the National Assembly. I think against my party. But I know that in the lectures.  The lawyer says no one has the constitutional power or is a kind of exclusive power to me and my colleagues to decide on our leaders. Am I doing any wrong when I say that I want them to choose me? I don't say wrong because I'm doing the exercise, I'm exercising the constitutional responsibility. So I'm not kicking anything against my party or against anyone. But I'm saying that I'm trying to get the indulgence of my colleagues to trust me and hire me. And at the same time I did the wrong somewhere, the hire me. So the hiring and hiring is the responsibility of me.


Can everybody hear me? All right. So first I want to start by asking how many of you actually are aware of what a distributed denial of service attack is? And how many of you have actually been hit by one? All right. We have people who know what it is. So the way the talk is going to proceed, I'm going to talk about what a distributed denial of service attack is, different kinds. And then I'll talk about mitigation. So first I'm going to talk about myself. Who am I? I'm an operations engineer at Flipkart. I've been doing the internet since 2005, and I've worked with a whole bunch of companies. If you want to reach me, I mean this is the LinkedIn address. So what is a D-ROS? I think this picture is a very good representation. I mean, if you want to consider the train as the service that you're hosting or a website, and all the people are trying to get on board. So the D-DOS can either cripple your service or cause a complete outage. What are the different kinds of attacks? First we have the volumetric attack. It's mostly an attack which is actually trying to consume all the bandwidth that you have. It is not necessarily causing an outage of your service.  Very common methods are DNS amplification, SNMP, NTP, and SYN floods, and all the fragmentation. DNS, as you know, is based on UDP. So one of the biggest challenges here is there is no connection that's established. Like there's no three-way handshake, et cetera. So an attacker, what he does is he spoof the victim's IP address and sends a request. So ISC is a very common target, not a common target, a place where the attacks originate from. So if you do a dig, I forget what the exact query is, the response that you get is about 3 MB in size. So the query that you send is about 100 bytes, and the response is 3 MB. So the attacker sends a request with, suppose, my IP address. So I get like 3 MB of data, which I never requested for. It does not hurt me in any way, but the thing is the request come to the firewall and not drop them. But they are choking the internet pipe. Second is the SYN flood. How many of you know you can stop it using SYN cookies? Or we'll likely suggest using SYN cookies. But the thing is, if it gets fairly large, you start having a problem with SYN cookies. Because you run out of the limited set that you have, and the server then decides to randomly start dropping sessions. And fragmented packets, well, these are a problem with firewalls or systems because they require extra computing power. The system has to reassemble the packet, and only then it can look at the content of the packet and can decide what it needs to do with it. Moving on to application layer attacks. Application layer attacks are slightly more complicated. Volume metrics was limited more to layer 4. Application is more towards layer 7. Recently, one of the attacks that we've seen is a WordPress ping back. I'm not 100% sure what it is, but from what I've gathered, it is a ping back from the WordPress server. So if you link a site on your WordPress, WordPress keeps sending a request, a ping back, to the address. So we saw a couple of thousand requests coming in recently, exploiting HTTP. This is a hybrid attack. In this, what we'd seen was we very aggressively compress HTTP responses. And we saw users had sent HTTP requests specifically with the compression disabled. So it caused a huge spike in our bandwidth. And I think we got to like 90%, 95% of our internet connection and the ISP start dropping packets. So not a good experience. Excuse me. The last one is incomplete request. So as you're aware, in HTTP 1.1, you need at least two parts of data. One is the actual request, like a get or a post. And the second is the host header. You can also have other details, like cookies, et cetera. But that's optional. So in this case, what an attacker does is he will send the first line of data. He sends the get URL, and then he'll just wait for some time before he actually sends the host header. So let's say he waits for like five or 10 seconds and he sends the host header. So in HTTP 1.1, you have to finish a request by having two new line characters. So I can send one and wait for another 10, 15 seconds before sending the last one. Now, the problem with this is it's consuming sessions at the web server. It's not causing any other problem. And this attack actually is a very sophisticated attack. In this, we had seen, I think, we seen a couple of 1,000 requests. And the increase in bandwidth was about less than an MB or close to an MB. So it's very hard to detect attack if you don't know what you're looking for. And it took us quite a while to actually figure out what was the problem. What are the steps for mitigation? So we'll start with the mitigation of volumetric attacks. For us, we own our own address space, IP address space, and we advertise it with different ISPs using BGP. So if, and this is actually a very specific use case, if the attack is localized to one particular ISP, in our case, we are multi-homed. And we are peering with more than one ISP. So if ISP is getting, if the attack is coming via ISPA, what I do is I will disable my advertisements there. This only works if the attack is not directed at me, but at the ISP. I mean, when you're establishing peering, you also have the ISPs address in there. The second is working with upstreams. Again, this is for large volumes.  An example of this could be, say we have an ISP, or a lot of requests are coming in from an ISP in Ukraine. We can have our upstream provider black hole us for Ukraine, but it's a little dicey because you might not just get blocked for Ukraine, but maybe the entire Europe. So there's a little gray area there. And the last one is use of scrubbing farms. So when all else fails, I mean, India has limited bandwidth connectivity, and the ISPs, they don't want to give it to you if you don't want to pay for it.   So in that case, what we do is we switch over to a scrubbing farm. This is basically a third party DDoS mitigation service. We change our DNS and let our traffic flow via the scrubbing farm, where they mitigate the attack and send it back to us. The other is application layer attack. So for this, we have two solutions. One is a homegrown solution. I'll talk about it more in detail. And the other is, again, falling back to the scrubbing farm. The homegrown solution that we have is, again, limited by the amount of bandwidth that we have, the capability. So if we can do up to medium levels of attacks, or like I talked about the incomplete request attack, these kind of attacks can be mitigated by us because they don't require a lot of bandwidth. So wherever bandwidth comes into play, we have to fall back to scrubbing farms. So before I actually talk about the solution that we have, I want to talk about why we built our own build versus buy. So we built our own because we know our application better than anybody else. We know what the flow is supposed to be, what the user's journey is through our website. So we are the best people. And we can identify malintents very, very quickly. Secondly, we want to understand the attack and evolve with it. But this, again, is subject to the complexity of the attack. At times, we've had the attacker using only one vector. So it's not really a sophisticated attack. I mean, it could be them trying to just do NTP. And once you take care of an NTP application, they stop. But at times, the moment you stop NTP, they hop onto a different vector. So in that case, we want to know exactly what is going on. And hence, we want to evolve with it. The third is the service providers have a very genetic solution. They all want to sell you an application firewall. Application firewalls are good, but they only work when you talk about XSS or SQL injection. They're not aware of how our application works. Most of them do not have enough intelligence to build a profile of my application. So it's not a feasible solution. So what do we do? We do real-time log analysis. We aggregate all our logs from all internet-facing applications and firewalls. Using LogStash, we get them all into Elasticsearch. And once they're there, we identify standard patterns, just things that we've looked in the past. We use those. And then we also work with the application. Every application does its own request profiling. So within the Flipkart website, you have a checkout or a commenting service or different services. They will profile the request and see if the user is following a pattern. I mean, if you are, suppose, just continuously searching on the Flipkart website, and we see 500 requests, you're not really a customer. Because you will search and actually go somewhere. 500 requests, just search request is not something that's a positive sign. So what we do with this is, once we have this, the applications give us feedback using a custom error code. So instead of, say, HTTP 403 or 404, we will use something like 999. And we are, again, doing real-time log analysis. So once we see that, we pick that data up, and we go ahead and block it on the firewall. I mean, there are a couple of steps here. We use different error codes. So you can either block it on the firewall, or if you want to start rate limiting it, we can do that as well. And the third option is the request profiler itself can present a capture to the customer. That actually happens in the case of people who are behind a NAT, mostly for offices or college campuses, et cetera, where we have a large number of genuine users who we do not want to block by blocking the IP address. What are the challenges that are currently associated with this? What we've seen is response times. In the last, I think it was about two months ago, we had a very interesting attack pattern. We saw attacks which would last for about five minutes. So by the time we got on VPN, we logged in. And to figure out what's going on, it was all gone. So it was a cat and mouse game. It was just very difficult. So we wanted to automate it. Hence, but not everything can be automated. So response time, again, is a challenge. The second is a maturity of mitigation solutions from ISPs. All the ISPs we've talked to, all the third-party mitigation providers we've talked to, everybody is reselling the same solution. It's basically an ArborBox. And they're just selling a subscription for that. So and the solution that they have actually works very well for layer four, but does not do anything at layer seven. And the other thing is even the Arbor solution that the ISPs have in India, they do not have enough knowledge about or hands-on experience with the DDoS mitigation. So I mean, it's not really effective. An example of this is we were getting hit by an NTP amplification attack. And the mitigation provider decided to go ahead and enable all defenses available in Arbor for our IP address. And one of the mitigations is also HTTP authentication using HG digest. That is to basically differentiate between a machine and a real user. So what happened was they, I mean, while trying to mitigate an attack, they actually started one because about, I think, 70% or 80% of our users just dropped off once that happened. Then the other problem is the Indian ISPs are not flexible. They are still operating in the past. I mean, like we have the cloud model where we can go swipe our credit card and we have machines ready and we have everything available. I can't have anything like that with the ISPs. I mean, I can't have them provision FAT pipes for me and I only pay for them when I use them. If they're provisioning a FAT pipe for me, they want me to pay for it regardless of whether or not I'm using it. So that is a very, the cost is a very big barrier. The fifth point is, yeah, fifth point is blackholing of our traffic. The ISPs are very quick in blackholing your IP address. If they've tried their mitigation solution, it's not helping. They've tried, their knowledge of DDoS is reaching a level where they can't do anything. And the attack on us is actually hurting our neighbor also. And I mean, even an attack as small as 10 GB hurts the neighbor. I mean, we've been victims and I'm sure we've affected other people also. So they're very quick in blackholing traffic and they do not inform. I mean, communication is a very big problem. At times, I think it was about 15 or 20 minutes later, we found out we had been blackholed. I mean, that was not a thing we were looking for. Later on, we became more active in monitoring our availability outside of India. And the last is there are no scrubbing farms in India. So when we utilize scrubbing farms, you have things like Cloudflare, Ultra DNS, or even Arbor has their own scrubbing farm. They're all located outside of India. In Flipkart, we very aggressively monitor end-user latency. Hence, we are multi-home. Hence, we actually decided to host in India versus going to Singapore or US or something like that. But once we go to scrubbing farms, the latencies are just out of the window. And there is no telling. At times, we've been told we are actually going to a scrubbing farm in Singapore. But again, it depends on the volume of the attack. The ISP sometimes does move to a scrubbing farm in Europe or America. What that actually does is introduce 2X latency for the end-user. And they have to first go and connect to the scrubbing farm. Then the scrubbing farm will actually proxy a request on their behalf to us, get the response, and then send it back to the user. So that's like 2X latency for the end-users. So these are the challenges that we've had. And if you have any questions. Hi. Can you throw some more light on scrubbing farms, as in what all services do these farms provide? The scrubbing farms are mostly black boxes. What they do is they're sort of reverse proxies. They also do pattern identification. And if you want to think of them, just think of it like a farm of NGNX servers or HA proxy servers, who will take your request in. And then you can apply filters there. So let's say if I identified a particular user agent that was bad, like in case of WordPress, we actually saw the WordPress appearing in the user agent. So they'll take that and start blocking it. You basically, with the DNS servers, you route all your flipkart.com traffic to them, and then they forward it to you? Yes. So I mean, that is how we move traffic. I mean, the other option being BGP, but the routing seen in Asia is very, very bad. Just to give an example, with one of the ISPs, we tried doing anycast. So we tried anycast in three locations, Delhi, Bombay, and Chennai. And what we saw was all the traffic just shifted to Chennai. I mean, even though I had a box right in Delhi, right next to the server, which was hosting the anycast, but it was all going to Chennai. So BGP is a very gray area, plus you also give away the control of your BGP advertisements. Right. So the thing is that DNS servers are something that are not in your control, right? No, but so we have a very low TTL. If you look up Flipkart.com, you have about 60 to 90 seconds. OK. Hello. We are currently contemplating on working with Arbor. Can you throw some lights on what have been your bad experience with Arbor? It's not with Arbor particularly. It's the people who are using that equipment. They don't know what is the capability of the equipment. It's like having a firewall. I can give you a firewall. But if you don't know how to use it, how to tweak it, it is not the capability of the firewall. It's the capability of the user that's actually limiting them. And also the other thing is, I think in India, the ISPs don't have enough capacity. I won't name the ISP, but one of them has about 5 to 10 GB of mitigation capacity, but it's advertised as about 100. I have a question about an API not about the websites. So I have an API which is, by nature, it's a public API because it provides, because it's part of a search as a service. And so now I am possibly thinking, how can it be misused by a DDoS attack? Because there is an authentication, in most cases, in that API. So even with authentication, you can't prevent against a DDoS. I mean, the authentication is going to be done by API. Or any other service for that matter. I mean, if we go back to this slide, there's actually nothing you can do here. The people are trying to all get on board. To mitigate it, what is the use of this for an API? How do you mitigate a DDoS attack for a public API? See, again, it's about muscling it out. In this case, if we have maybe 10 more trains or a longer train with more seating capacity, that is the only way you can solve it. But there is a limit, right? So let's say if you're hoping for 100 requests per second, you can plan for maybe 1,000. But if you start planning for a million to mitigate it, I mean, there's going to be a significant cost associated with it. So you could take help of people like Cloudflare or Ultra or Arbor by itself. It's on the same lines. You mentioned about homegrown. Application solution for this, right? What kind of solution was it? And then you had some points about logs doing more analysis. But what is specific in the application that was done? Like I mentioned, it's our application. So we know it best. So we're aggregating all logs from all web servers or actually all internet-facing applications. So basically logging as much activity and then being proactive enough to know that there's a pattern coming. So there are some patterns we know of. There's some patterns we can actually look at and start investigating. We are not there at the artificial intelligence level to automatically start flocking stuff. But the whole thing with that is with, so let's say you have an API for Flipkart. You have a seller platform for Flipkart Marketplace. And you have the customer-facing website. Now, if I see an attack on the seller side, I can take that information back and actually start blocking or take preventive actions on the customer side before it actually starts happening. So it's a global feedback system. So from all applications, I have that feedback. And then once I've blocked them on the firewall, so what I've done is the firewall is sitting at the edge of the network. So once I block it there, if I still see traffic from the user, I'm getting my firewall logs as well, I do not unblock them. I mean, as being an internet company, we unblock our users after x amount of time. And that's very small, like 10, 15 minutes sort of thing. So we had that feedback loop that even after being blocked, if you're still attacking us, we will not unblock you. So it's not totally a preventive solution. There'll be some initial attack that will come in for us to get that data to identify and then build on. So for the home room solution, you have your own HTTP protocol decoders and everything in place? No, no, no. We are doing log aggregation analysis. That's all we're doing. So we're using LogStash, and then we are putting all that data into Elasticsearch. So you can compare it to a Splunk. So when did you understood the first attack pattern?  How did you started doing the solutioning? I think it was the third or fourth time. In the last four months is when we've seen a rise in the number of attacks that we had. The first attack we actually saw was in 2011. And after that, the attack started in December. And as a whole, they've started increasing. So with the amplification attacks, there wasn't anything we could do.  In terms of the application layer attack, so we have graphs for everything. The QPS, they were doing the amount of data that we were pushing out per request, not per request, like through the website, and also at the network site. So we saw spikes in bandwidth. So we started investigating. We also saw spike in, not really a spike, but an increase in the number of incoming requests. So first thing was, we went and checked with marketing if they had something going on. But we didn't really find anything there. It took about, I think, two, three hours to actually find out what the problem was. So very interesting thing. I mean, Nginx also has a thing called a lingering time out. I mean, that is when we learned about it, where if the request doesn't come in within five seconds, which is the default value, it drops the request. So we saw this pattern that the request for dropping after five seconds, but we just had no clue why. In Loctash, you must be capturing HTTP logs. What about other attacks like NTP or DNS, for that? See, NTP, I don't have a port open on the firewall. I'm not hosting a server. But the thing is, it's choking my internet pipe. Even if I'm dropping it, I'm dropping it at my doorstep. But can you capture the logs for that? I have firewall logs. So you are capturing the firewall logs. So the net screen, they have the number of packets per IP and protocol matching. Aren't you compromising your security by describing so many of the protections, checks, that you are doing? No, I actually have a very generic presentation. Thank you.


Honebo, Muniniazu, HMP, Fulumezi, Wanonwanonbuba, Army of Hosts, Frank Mutubila, Mao Samba. Welcome. Thank you, Meseboku, Ziko. Tell us. Mwachokema kutibaba? No, Tachikapa Shimbokaira, Mausaka Central Prison, where I was being kept.  Why? Tell me politically. Politikowa is unmotivated because I used the same bathrooms that the current president used when he was in detention. I mean, it's a privilege to use the same facilities as the head of state. It means you're also being baked to be... I'm heading in that direction. Which direction? Kwazi. I'm going to Kwazi. Statehouse? Statehouse. Tell us. So don't worry when you hear me declare. I know you declare early, but I think I'm joining the rest. The more the better. Yeah, I'm joining the rest and I only try once.  No. So... Either papati or independent? No, I just go once. You don't lose. And get it. That's the only thing common between you and me.  If you don't be like that, just be a man, test the prisons and then before you take it. So my CV is complete.  No one knew that your own brother. Some people even think you are a young brother of Kaiza. We used to think Kaiza is because you share the same surname. Yeah, but for your own young brother, why would you do something like that? You see, it's like the store of the UPND using the PA for the benchmark in everything they're doing. So don't use my case to be a benchmark. It doesn't mean that if Kaiza is an irresponsible human being, then others are also irresponsible. But if they're human beings and Kaiza, so Kaiza should not be used as a benchmark. We did everything in good faith. We thought he's no more human being, but here we are. Let's be honest. I'm one of the assurities and obviously we know they cannot run away. For all we can say about them, but they cannot run away from the court. Nangam fana wu muche omuzi wanae wu. Nangam bena enda manchizangala wangi. At least I cannot... Manchizangala wangi. Manchizangala wangi. Everyone was refusing. He was a man for two months. I want you to take it from me. There is a greater God up. We are not doing these things because we want to be trained. We are doing it out of being human. Go to judge. Akata wa nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Nivaku. Naikaan, mahiina mahiina classmates ka hita ni oliiwa dola package zumolo no Masa na kasha matun So, let's see he, he let, he that bigapsu He did the bigapsu ja bigapsu nama analysi Bata ni na tu na mudin Enem a no mo yuni once imagine hegata su sta Oro tab Surface not Bata ni kuna apati nsmuka he Miki Shannon Kanata Miki Maiki Alama.. Mimungamiya siyasuchw ka,kwenando Kwanando unibusi Seza kuwa kuwasia kwenando Uporta Niru Uporta Rabi Hangi onion gaibimpu Uotima Sh Unfortunately Uhtize I think they should expect more of these difficulties. And what's your message to Zambians? Change is coming. The younger generation is now ready to take charge of their own affairs. So change is coming. I think by 2026 we are giving them a president of one generation. How old is this?  I'm a consultant. We have been working and working. We have been giving advice and advice. We've been giving advice to the younger generation. We're giving advice to the older generation. T shaped Fire Fire Fire Fire Injiniya, Azangara, politiko advisor wa mu. Inesdam sa udara wa fachia, Azangara national youth chairman wang. Manji o national youth chairman no politiko advisor, which is which now. So already you're undermining the engineer. He needs to be with the president. How's your family since you've back? Have you seen them? No, they're fine. Mwenza kutia chanu? Mwenza kutia chanu? Mwenza kutia chanu? Mwenza kutia chanu? Mwenza kutia chanu? Mwenza kutia chanu? Mwenza kutia chanu? Mwenza na wa? Mwenza kutia chanu? Mwenza na wa? Mwenza na wa? Mwenza na wa? Mwenza na wa? Mwenza na wa? Kama Inesda Naction. Lawakali na wa wowa? sutukuwe. Isaving theBoaz tiring hard to go ano? Tawu wa yogo muchi ko? Kumarasa ni m telescopei n 진akumiko. Korezuna wang village of sa n작asun. We have to go fasting for 7 days for the storm. Good to see you Mwana. I am debating this afternoon. I was missing you actually. I am debating this afternoon. You heard the story. That happened when you were inside. We keep coming our prayers and we find time to visit. Amongst my closest in parliament, D-way and you know I was a guy. Most unfortunately I wish him well that he has been evacuated today. He has? Ok then, what time and visit? He has been taken to S.A. That's just one hour forty five minutes. What time and visit? You look smart. You know we have any problem. The generation president? Ok, why? We have to go fasting.


Live from New York, it's theCUBE covering big data NYC 2015. Brought to you by Hortonworks, IBM, EMC, and Pivotal. Now your host, John Furrier at George Gilbert. Okay, welcome back everyone. You're watching Silicon Angles theCUBE, our flagship program. We go out to the events and extract the similar noise. I'm John Furrier, the founder of Silicon Angles, joining my co-host George Gilbert. We are live in New York City, 100 yards away from the Javits Center where Strata Hadoop is going on. In conjunction with Big Data NYC, our event we're here for three days of wall-to-wall coverage, getting all the data and sharing that data or putting a data pipeline of content together for you, covering all what's happening here in New York City. Our next guest is Rishi Yadav, as the CEO of InfoObjects, CUBE alum, and now published author of his new famous book. It's sold out on New York Times bestseller list. Welcome to the CUBE. I'm only kidding about the New York Times bestseller list, but let's chat. So congratulations on publishing the book. Last time we spoke, we had the cover, and it was cover on another book, but it was a good prop for the camera. Today we don't have the book, because they're all at the booth getting signed. Give us the update on the book title, how to get it, URL, and comment. Yeah, so the book is Spark Cook book. It got published five, six weeks back. It's getting amazing traction. Yeah, you can go to Amazon or Bonson Noble, so any usual place and you'll find it. So what was the big thing? So let's just revisit our story, because we've been well-documented on the CUBE, the interviews you can go search at youtube.com, slash look on the angle on InfoObjects, look at the videos. Going back three years ago when we chatted, it was not obvious to everyone in the industry as it is today. I mean, they should call this, rename this event from Strata Hadoop to Spark World, because all the talk is Spark, Spark, Spark. You know some ancillary announcements, but you were on the front end of that curve. Tell the story and are you happy you made the bet and what are some of the outcomes that you guys are doing today from a product standpoint and then customer benefits? Yeah, so we took a big bet on Spark and that's been a blessing for the company. I mean, for a consulting company, choosing one path and then getting all the inbound client leads. I mean, that was the biggest proof for us that it's working. I mean, before Spark, we focused on Hadoop in general and there we had to struggle to get clients and the moment we focused on Spark, the business has been really, really good. So I saw Merv Adrian had a tweet from Gardner just earlier today saying, finally everyone at Hadoop is talking about data in motion. That really is kind of categorically discussing real time. This is some of the real time analytics pieces that you guys are focused on. How far along has the industry come with respect to Spark? Certainly it's on the radar of everybody. We've got some success companies being developed. What are you guys seeing in terms of Spark, VisaV, Hadoop and how it all fits with moving the ball down the field faster innovation? So as far as Hadoop is concerned, the Hadoop store is the HDFS, right? I mean, that's become a defective standard in the industry. In fact, HDFS I would say is also eating the lunch of HBS and Cassandra's of the world. Because what's happening is with the new file formats like Parquet and all, now HDFS, what HDFS did not have was the schema, right? And now with the file format like Parquet and what's happening is that now it's as good as having a database, right? So you have all the distributed power of the database, right? For scans, in other words, like MPP type scan and aggregate. But the HBS and Cassandra were more oriented towards point lookups and simple transactions, no? You know, where HDFS was kind of a different end of the spectrum. Yeah, so HDFS, the problem with HDFS was that you could put anything and everything there, right? Which was a good thing, right? As far as putting the stuff was concerned, as well as data ingress was concerned. But when it comes to taking the data out, right? When you're trying to read the data at that time, you have to figure out that what type of format you want to put the data in and that had its own challenges, right? So schema being separate and data being separate. What has happened with Parquet and Orc format and a couple of new formats coming in that is that now one file will have its data, the file will also have a schema, right? And in Spark, at the same time, they moved from the RDD to a data frame, right? So data frame also has data in it and it also has schema in it, right? So now in one simple call, you can read the data and load the data the way everybody's used to loading the data that is in the case of databases, right? So, but database is a very, very small subset. Here, you're talking about petabytes of data, right? So now you can access petabytes of the data the way you are used to accessing in the databases and at the same low latency as you are used to accessing. So the paradigm that we're seeing, I want to get your thoughts on this, George. I know you got some more questions and hold on for a second. I want to get the perspective here from the expert. The old model was put everything in Hadoop. You saw that, made some money, did some good business with customers and then realized probably what we're seeing even today. Good call, by the way, because Gartner's also saying that POCs are up but productions aren't as heavy, so you had good decision to go with Spark. But you move the data into Hadoop and you move compute to the data. That's where storage has got value, hence the Cloudera announcement. But now the big trend is pipelining, getting into the flow and using real time machine learning, being access to that, trapping on data and learning about the data, the machine data. Machine speed is faster than human speed, so there's a new streaming, and we hear all these things. Talk about what this is going on. Can you share your thoughts and color around the trend of, not the Hadoop putting compute to the data, but happening in line, in the memory, in the data, in the flow? Yeah, so we, it would not be right to talk about the streaming part without talking about the IoTPs, right? So IoTPs. So one thing is IoT is that you see it and all the sensors and bulbs and all, right? But the biggest IoT piece is in the industrial IoT, right?  Industrial IoT. Machine data. Machine data, right? Yeah, now with the power of Spark and the streaming and all this machine data, you can get in real time and then do online machine learning on that, right? Yeah, so you're right, John, on one side we just were storing data using Hadoop and doing some analytics using Spark here and there. But the biggest power now has come is that now in the real time, you can stream all the data from various locations, thousands of locations, hundreds of thousands of locations and do online machine learning on that. What is this doing to the landscape for application developers? Meaning more catalog driven, is it much more SOA? What is the impact of the developers, this new trend? So application development actually has got simpler. Two, three years back, if you're developing something in Hadoop, you have to be a Java developer, right? Then in Spark you have to be a Scala developer and now all that has become what is it in the database industry that you just need to know SQL, right? So most of that has become just putting SQL jobs. So the whole application development landscape in the big data, actually that has become easier, simpler and narrower. I got to ask you a question. So we were having a crowd chat earlier, crowdchat.net slash strata Hadoop and one of the most voted up threads was a comment from Renee Yao, she commented, Gartner mentioned 41% of organizations don't know of big data. ROI will be positive or negative thoughts. She put a chart up there. Essentially, the vendor community, it's all about ROI to the customers. What's your thoughts on that? Are customers scratching their heads with ROI? Are they still in the low-hanging fruit stage? And if so, what is that low-hanging fruit? Talk about this dynamic. We're almost half of the organizations don't know if big data ROI is positive or negative. What is that? Do you believe in that or what's your thoughts on that statement? So what I see on the street is completely different. What I see is no client has any doubt about the ROI of big data. The challenge they have is that how do we get that ROI using Hadoop and Spark and other technologies? And what they're doing is on one side, as I talked about IoT piece, the other big piece is the enterprise data piece. So Hadoop being unlimited storage, so there the enterprise data which used to go on tapes now is going in Hadoop. That's number one. But at the same time, a lot of compute also is moving from the enterprise data systems to the Hadoop systems and Hadoop and Spark-based big data systems. All right, so I want to ask you a question about Spark. What is your biggest learning since we've last talked in the Spark community, Spark technology, customer adoption, all of the above? What is the big realization that you've come to at Info-Objects and also you as an individual? I think what we talked last time was the data frame which I mean it's just a technical piece but what it has done is that it has made a Spark object like any other table, right? And that made it very easy. So now you can have the same data in the enterprise data store and the same data you can have in the HDFS and using Spark on one side and let's say SAP HANA on the other side, right? To be clear, the data frame is like the equivalent of a table, so you have a table data type that works across all the Spark APIs and it works across all your familiar tools outside the Spark ecosystem. Exactly. So I got to ask you, you have a blog post on your site that's very impressive. We oxygenate the ecosystem. Explain what that is and the motivation behind that post because since past decade that you've been involved in the industry here. What does that mean and what does that mean for the future? Right, so I mean companies like InfoObjects, we have a unique role to play here. I mean on one side you see Hadoop vendors like Cloudera Mapper and Hortonworks, right? On the other side you see big consulting companies like Accenture and all, right? I mean do it all kind of companies, right? And we go as a trusted advisor to the client, right? And we go to the client and we tell them that in a vendor neutral way that for you, I think adopting this big data strategy would get you the most results, right? So we are kind of filling this gap here, right? In which on one side client is looking at their ROI, the Hadoop vendors, they are looking at more their stack perspective that what their technology stack has to offer. Just to clarify, when you go into a customer, do they bring you in to advise on a big data strategy or do they bring you in to say look, we've got this application that we really want to start making use of near real time data, generated from machines and we want to be able to create a pipeline that has a really fast feedback loop for improving predictions or recommendations. Where in the spectrum do you come in? So when we started in Spark, most of the clients would come to us for troubleshooting that I already have a cluster and somehow it's not working. How can you help us optimize it? What I see more and more is as companies are getting deeper and deeper into Spark, they are involving consulting companies like us early on, right? So now companies are involving us from the advisory phase itself, right? Okay, I have to have a big data cluster here, but let's help us architecting that cluster, right? So, and that helps us a lot because then what happens is that all the learnings which we have from implementing it for other clients that we can implement early on. So they don't have to repeat the same mistakes which other clients did. You bring up something that's interesting. You talked about also in your blog post, throwing bodies at a problem, these big consulting shops, whether it's outsourcing or whatnot, they weren't really working the solution.  And Jim McHugh from Cisco said on the Cube earlier today that a lot of customers have been vendor hopping. It's kind of like party hopping. You go to one house, then the next one, mainly to test, oh, they got this version of Hadoop, during that early exploration stage. So that party hopping, kind of vendor hopping, we're seeing people out of that phase, at least for the most part. There's still that going on, but for the most part, they kind of it's tasting around. What does that mean for the customer who's looking at it? Hey, I've tried a little bit of Hortonworks, I've tried some Cladera, I got some EMC, I got the Pivotal. They're like, okay, they need to step back. What do you guys say to that customer? What is the main message? Yeah, so it becomes a challenge and customer, when they do vendor hopping, they are not happy about it, right? Because they say that this vendor hopping is happening in our time at our cost, right? If we already knew that what right strategy we have to follow for our needs, we would not even have to get involved into that part, right? And that's where info objects come as a trusted advisor, as I said, that now more and more clients are involving us early on, right? And at that time, we always look at what client wants, right? We do not even care about what stack a specific vendor has. Sometimes we can even combine the stacks to look at the client needs, right? So that's where we come and we are completely vendor neutral, I mean, we do not represent or resell any vendor solutions, right? So that also helps clients a lot, that in fact what's happening is, say a client may be expert in their own industry and now because they are partnering with us, now they've also become big data expert, right? Now loaded with this expertise when they attack the big data challenge, right? Their strategy is much more successful. So what is the big takeaway for you this year at Strata Hadoop? When you look at the ecosystem, the floor, the vibe, what's your take on what's happening? So it's too early, so I mean, we are just setting up booths today. So maybe in a day or two, I would have a better perspective about it, but the floor looks much more vibrant right here. I mean, I see a lot more shops here than I saw at Hadoop for a couple of months back. So I look forward to see what they have got. How about the vibe? What's the vibe of the show? I mean, is it sold out, crowded? What's the vibe? I found it fairly crowded today, so. Let me ask, I'm curious, we talked about how when you first started coming in to cluster engagements early on in Spark and it was troubleshooting and now it's more and more advisory. Sort of, how have the problems changed? The customer use cases changed and what are they talking about? Not just what are they doing now, but when they talk about their plans two, three years down the road, what do those look like? So the big data lake, I know this term has been overused, but now actually, in fact, I wrote in one of my blogs that


This paper proposes an ontology to facilitate the identification of patterns of damage from recorded image data. The ontology will link information items such as earthquake magnitude and distance from the epicenter to the extent of damage caused by an earthquake. Additionally, special tools will be used to select sub-mobiles related to structural damage and identify each type of damage. This will enable faster processing of large volumes of image data collected from heritage sites. This article was authored by S. Ryle and H. Essel.