Welcome back everyone. Once again, I'm Austin Bingham from 60 North up in Norway And I want to present to you Konark Modi who will be talking about near real-time stream processing in Python with Storm and Kafka take away. Hi. Good morning all of you I understand I'm the only one standing between you and the lunch So I'll try and make it interesting for you guys. All right So the title of my talk is designing near real-time stream processing systems now What do we mean by data pipelines and near real-time systems? So everybody of us knows that in real time we are generating a lot of informations as and when we visit websites We browse a lot of applications and stuff like that And we need reliable and quick mechanisms to collect that data Once we are able to collect that data. We need to process and analyze that data as well, right now Systems like Hadoop let you efficiently do that But that's again batch processing system when we talk about batch processing systems We mean that there's going to be high latency of updates in your query results And when you have an application where you want to provide recommendations You want to provide better features to users you want to provide a feedback loop mechanism You want a layer which helps you work with the recent data and update with low latency of update time So that is where the essence of near real-time data pipelines would come How I treat data pipelines is majorly in three parts one is the input that is basically the messaging layer Where you're collecting all your user user data that is coming on your streams Second is the processing layer where you will be processing the recent data Okay, now when I say recent it defines say in n minutes or n days So it's a defined moment of data that you're going to process on your near real-time stream processing data I'm not going to process one months of data on my near real-time processing layer, right? And then when I process that data there has to be a layer where I can present my data in Probably a query will format or a format that my other applications can consume in a feedback loop. So today's talk Is majorly focused on two components one is the messaging layer That's Kafka that I'm going to take in the second part of the talk and before that we're going to talk about Open-source project which was open source by Twitter is now a Apache project known as Apache storm Primarily Apache storm is 50% closure and 50% Java But that does not stop us from using it in Python And that is why I'm here today to present how we can use Tom and its capabilities full-time using Python itself So how I try to structure the talk is Just before that can I have a show of hands? How many of us actually know what storm is? Cool, and how many of us know what Kafka is Right, so we have a majority of people that are still not known to storm in Kafka So I'll just give a brief of what storm is and what Kafka is Given the time frame and the kind of content that I have I might speed up speed up a little bit But do feel free to catch me after the talk and I'll be happy to show you demos and other stuff All right, so When we talk about stream processing there are a lot of challenges that you need to tackle But this slide only fits a few of them first of all For me storm the a near real-time processing system is actually an infrastructure that will do a never-ending data processing for me All right, so what I mean by that is it's not like a map reduce job that at a given point of time It will finish right a real-time processing system for me is it data keeps on coming in huge volume It will keep processing it and keep pushing it ahead, right? It will never end for me So that is what I mean by real neat real-time data processing pipelines, right? You have to store messages somewhere before your consumers can consume it You want you might want to replay your messages again. You might want to drop few messages and stuff like that So all these things need to be taken care by your real near real-time processing layers You need to route your messages for example, you're collecting few messages from application one You're collecting few messages from application two you want your all your application one messages to route to one particular worker Application two's message to route to particular two different worker So these are few nuances that revolve around a real-time data pipeline processing system, right? You want to scale for high throughput for example, you design a system that is taking care of thousands of messages today But tomorrow you might scale and you might want to use a system Which will then process about hundred thousands of message per second, right? So you need a design philosophy around your system that takes care of all that When we talk about storm it it takes care of almost all these things for example You use Tom and then you have Already solved a lot of variety of challenges that you can say for example You just want to process all the Twitter feed that you're getting from fire hose and dump it to your DB All right now you in systems like this you need really need to take care of the back pressure for example What if my processing layer dies where do all my messages go right that are coming from the Twitter hose or Finally my persistent layer is down. So how do I store my position layer or how does the Processing layer take care of the layer ahead what happens to those messages, right? So Twitter storm a patch of storm. Let's you handle extremely Broadcase of scenarios for example, you want to do a query on fly and again That has to be distributed across because your data is distributed across or it takes a lot of time, right? So a patch of storm has something known as DRPC topologies that helps you do that as well It is scalable. So when we go in the architecture of storm, you'll understand how what we mean by scalable in terms of a batch of storm So you have different services that do different Tasks in storm and that lets you scale as well when it when we talk about scalability We obviously talk about fault tolerance as well What happens to one of the workers if it goes down what happens to one of the services that coordinates all the workers go Downs as I mentioned earlier. I at any given point of time I do not want my data processing layer to go down, right? One of the components might go down they might come up But in the end, I never want the whole system to crash down for me because that is certainly not feasible It is program like language agnostic. That is how we are able to use Python with the patch of storm It supports all other languages as well So as I mentioned the core is in closure and Java, but it has multi-lang APIs that help you achieve a lot of stuff There are few advanced level of Mechanisms that we are still not able to use it Python, but I believe in their roadmap They are planning to open that up using the multi-lang API as well Achieving parallelism across components. So when we understand the various components of storm I'll brief you about how at each level we can configure what kind of parallelism do we need So these are the storm components I have tried and depicted them into two parts one is the conceptual view This conceptual view helps you when you're actually programming Your system for storm and then is the physical view. These are the actual services that will be running on on on your service on your on your systems, right? So first it's spout. Okay, treat spout as an ingestion layer. It's an ingestion component So spout is your interface to the outer world, right? So you write a spout that will basically interface with say Twitter fire hose or a rabbit MQ layer or a Kafka layer or anything like that and then spout basically generates stream Which we're going to talk about so stream is nothing but a unbounded sequence of tuples that basically go to bolts and other other components, right? So here in the example what I've tried and showed is a small Code snippet of spout Now it's doing nothing in the real world you will never come across the scenario because what's it's doing? It's not connecting the outer world It's just generating random sentences within itself and emitting them ahead for the further processing layers, right? So what it's actually doing is it says I want to generate random sentences from the list of sentences that I have and just emit each sentence, right? Streams we've talked about so If you can see in the code snippet what it shows is I'm emitting each sentence over here, right? So stream is non-bounded sequence of tuples so each sentence is part of the stream then So each message in the stream is the emit message that is coming from the spout Now these streams then communicate with bolt. So, okay There's a fine gray line, which you cannot see so basically bolt can have N number of inputs and N number of outputs So when we talk about inputs treat stream as an input to a bolt, okay? So a bolt can take an input from a spout a bolt can take an input from another spout as well a bolt Bolt's output can be input to another bolt a bolt's output can be a persistent store or anything else or a bolt might not just Emit anything it might be like a dead end for your topology, okay? How we knit all these together is what we call topology so in a topology you will have multiple spouts You'll have multiple boats doing a lot of stuff So as I mentioned spout is just a interface for outer world to get data into your data into your Twitter storm, right?  So bolt is essentially that the whole architecture when we knit it together We call it a topology. So to your storm cluster you always submit a topology, right now The flexibility is such that your topology can be a mixture of all technologies All right, so you might write a spout that is written in Java You might write a lot of bolts that are written in Python and that's really helps you a lot because for example in my environment We are a mixture of Java and Python devs, right? So I would obviously not rewrite something that is already written in Java or those guys might not rewrite what already has been written in Python so defining a topology helps me do that How we define this topology Now there are multiple ways to do that because there are multiple libraries in Python that helps you connect with storm This is an example of stream pass library that I'm going to showcase throughout the talk today So this is actually a closure DSL. That's written that helps me define my Python spout and my bolts So what what is happening over here is I define that I have a word spout So this is actually a bolt which is taking input from word spout. Okay, we'll get into the meaning of what shuffle is and I've told this bolt that the program that you need to need to run when you receive an input is the sentence splitter So what it's doing is it's receiving sentences and it's splitting the sentences into words The next bolt receives the input from sentence splitter, right? Which essentially it's getting the words from there and over here. We've asked it to group by words We'll come to groupings what groupings mean and Then the word count saver is essentially doing nothing It's taking inputs from the previous bolt, which is the word count bolt and then processing it as I mentioned It's a dead end. It's the last bolt in my topology. It need not emit anything You can emit make it emit anything or you need not make it emit anything, right? So this is what's happening in a basic topology So what do we mean by groupings now? so groupings Will help you tackle challenges like you want to group by a particular user and then process its data across one worker Itself because it's a distributed system if your data gets distributed you might have problem doing aggregates, right? So take a case of simple word count example now You're your spout is emitting a lot of words and your bolt is processing them and at the end of it you want counts of Counts by words, right? Now one way to do that is you have a persistent layer use dumping everything over there and that's basically say a red is INCR mechanism So you're pushing words over there and the INCR mechanism will take care of that in in terms of both What I'll do is I'll group them by words on the bolt itself So it makes sure that each word same word goes on to the same task and there I'll be maintaining aggregates for that The other way is shuffle shuffle make sure that each board to see if same number of same number of Streams, but they might shuffle across in terms of they are not grouped by any any fundamental key And there are a lot of other grouping mechanisms also. So the documentation has the complete list with good examples as well Coming back to the physical view In physical view, I'm sorry about the size of the diagram. So on the extreme left, you see nimbus So nimbus if you're coming from a Hadoop world, so nimbus is just like a job tracker What you do with nimbus is you submit jobs to nimbus nimbus make sure that these topologies are then submit to the workers. All right so Nimbus will take care of managing and monitoring your topologies around the cluster when you submit a new topology It takes care of the deployment when a task is assigned or in case of failure of a task It takes care of the reassignment. Now in the second slide, I had mentioned fault tolerance Now here comes the first part if my nimbus goes down my topologies will still keep running What what will get affected is the new topologies will we will not be able to submit the new topologies But the topologies already running will not get affected by that So so even if your nimbus server is down for say to two or three hours, your topology will still run intact, right? Zookeeper is the coordinator between supervisor and the nimbus supervisors are the actual services running on the workers that takes care of the topologies Now nimbus and supervisors communicate via zookeepers zookeeper is nothing It's a it's a cluster configuration tool that helps you manage what cluster stats are so that nimbus and Supervisors can be in sync again if my supervisor goes down So supervisor again has few parts like workers which has tasks and executors So if my supervisor goes down it will make sure that another worker is spawned So in that ways fault tolerance is maintained at each and every layer, right? Coming to the main part how storm and python fit in together first is stream pass. That's the library I'm going to talk about today. It's a library open-sourced by Parsley recently it got open-sourced. We're going to talk about stream pass in depth today stream pass Let's you write your topology. Let's you write spouts and bolts in python But you have to define your topology final definition of topology using a closure DSL The advantage of that is you can then mix multiple languages for example Java spout and python bolts So that is why I prefer writing that in closure DSL But if you want to write anything in pure python, there's a library known as petrol That was open-sourced by air sage last year. So that helps you write complete topology spouts bolts everything in python Now we have another talk on 25th, which I believe is going to talk about integration using jython So Jim is the guy who's going to be talking about that So he has a library called clamps that basically takes care of mixing your Java and your python stuff So that's even an interesting library. So I think you should visit that talk as well and the native Java Project of storm itself lets you write topologies. So but it will not let you define a topology in python. You have to write Your topology in Java. You can write bolts in python There are few implementation where people have written spouts in python, but that or not that's a successful So that's the last option. I would say you you can use Why I love stream pass. So this is basically the architecture diagram of stream pass if colors are visible to you So all the gray part is what is storm cluster, right? Now what stream pass let me do is it lets me create an environment around my whole storm cluster All right, so I am on my dev machine and I'm creating a storm topology, right? Now I'm defining bolts. I'm defining spouts What I'm what I need to do is sparse S-Parse is the utility that comes with the stream pass. All right So when I say S-Parse quick start a particular project, it gives me this beautiful project layout, right now each Folder and each file has a significance for S-Pars. So what it lets me do is it lets me Configure all my workers why I need workers is basically when you submit a storm topology via S-Pars via stream pass It basically does a SSH to each of your worker and deploys your python environment over there So for example, if I'm using something like WebSocket client or Kafka client that means it's external dependency that does not come with native python And this will make sure that it first goes on to the worker node and creates a virtual environment over there Where it will install all my dependencies, right? How I define my dependencies are in one of the files wherein you can say that these are all my dependencies So it's going to be in virtual environments and you create a requirements.txt file over there Right topologies is where you define your closure DSL So topology slash whatever name you want to keep for your topology you define your topology over there This is an example of a topology so it's it's a word count.clj file So what it does is it's tell that I have a spout which is words.py I have a bolt which is word count.py. It's as simple as that what you see in the bottom bottom row is p2 which Essentially defines the parallelism that your bolt needs to perform with Enough of talking. I have built a very small application around this which I would like to showcase so the application is built around Wikipedia edit trends what is happening at Wikipedia whenever somebody edits an article they release the edit log for that So it's real-time What what I've done is there are a lot of tools available to capture that feed from Wikipedia So I've just taken one of the tools it gives me a fine JSON message which looks like this. So basically we're going to trend three matrices out of it First is going to be action So when you edit an article on Wikipedia you can have multiple actions edit delete create update etc It tells you whether the article was edited by a bot or not So is bot false is bot true Whether the user was logged in or not while making the changes or not in case the user is logged in You have a username if not you you get an IP over here, okay? So we'll turn three matrices over here one is going to be action-wise trends So how many deletes how many edits how many updates then it's going to be humans versus bots a graph about bar chart humans versus Bots and last is going to be logged in users versus anonymous users. All right Now I'm not parsing the real-time logs because I were not sure about the internet connectivity So I've taken a subset of logs from From the Wikipedia feed. I have dumped it on to the Kafka Layer and I'm going to use Kafka for that after this we're going to come to what Kafka is all about. All right So what I did was I just did sparse quick start Wikipedia edit logs underscore trains It gave me that beautiful project layout that I was talking about After that, this is the product layout that I was talking about all the files are there Now what I do is I define a very simple Kafka spot. I mean there's nothing Kafka about it It's simple pub sub that I'm using currently over here What it's doing is it's reading a particular topic and getting all the log files and emitting that message, right? So when we say message over here, it's essentially a Jason log that's getting generated Now this bolt basically parses my Jason. Okay, so what it does is It checks what kind of action there is it checks whether the bot is human or not it checks whether users logged in or not and Stream pass let's me emit in form of batches So I would say emit underscore many and list of words is and is a list of list. So basically one output of this would be Human g2 underscore edit comma logged in or not logged in. Okay Now I'm appending keys like g2 and g3 because I'm not that good with JavaScript So that helps me at the web interface part of it. All right, so please bear bear with me for that So this is how I counted now whatever what I've done over here is I have a web socket server running when I count that I Dump the whole dictionary onto my web socket and that lets me plot my D3 graphs on the web interface So that's the simple final word counter board that I have for this particular example This is the DSL over here I'm saying you have to use Kafka spout then forward the outputs of the car first out to the past Jason and Once the past Jason has done his task past its output to the count port. All right So this is what it looks like after parsing. So this is a real-time parsing that is going on So this is the final output on my web socket server that is coming in and these are the graphs that I was talking about So first one is it's create edit and I think last one is delete Then you have bot versus humans and finally you'll have logged in versus not logged in users So the whole point of this demo is that it's pretty easy to get started with storm It's very easy to write your spouts and bolts and finally you have a real-time application that is coming in and it does not take much of Your task to manage the cluster as well one of the one of the headaches that come with systems like Hadoop is you have to do a lot of Cluster management with that but with storm systems like storm you do not really have to do a lot of cluster management And because we know that once a topology is running it will never stop how it will only stop is I have to manually kill that topology Otherwise it will always be in the running mode So as as in when the logs are coming they will get get parsed and that is what the meaning of real-time data pipelines at least for me is So this is how you run it. So when you do a sparse run It starts a local instance for your storm cluster to test your topology right there on in system Once you've tested it and you're sure that this works pretty fine for you You can do a sparse submit and name your topology what it will do is Now all the topologies get submitted only to the nimbus nimbus takes care of shipping your Code to the workers But in case of s-pars first it will SSH on each of your worker nodes because we need to create a virtual environment for python That basically takes care of all your dependency So that's an extra step that s-pars will do otherwise It's similar if you submit a Java topology or a python topology because it gets shipped to your nimbus via thrift Coming to Kafka Why do we need systems like Kafka? I mean we've got we've already got so many messaging layer around we've got a lot of databases We've got a lot of caching solutions around so where does Kafka Kafka fit in and what is Kafka? So for me Kafka is a high throughput distributed persistent messaging system that takes care of pubs up as well That takes care of high throughput that takes care of distributed storing off my data as well Now in the earlier challenges that I were mentioning I the first point and made was I need a robust message layer that Can that takes care of the messages until they are consumed all right and also once they are consumed I still want to retain those messages because there might be other consumers that want to consume them So when we're talking about a distributed processing layer there might be n number of consumers that want to consume the same message Right, so consider a topic for example wikipedia edit trends topic now One consumer is reading those topics and there might be another consumer that is dumping all those log files to my Hadoop layer Right, so what you need is you want multiple consumptions of the same messages You don't want all your messages to be on one box for say failover and high availability of a data You want your data to be partitioned across multiple clusters So Kafka would let you do all that stuff along with that it lets you maintain high throughput So you can have high number of rights and high number of freeds How it lets you do that is because Kafka traditionally is basically a file file system oriented queue So it persists everything on disk all right and by design it persists everything on disk It's not an add-on feature that is enabled disabled by default and you switch it on to persist message on the on the hard disk So basically Kafka will store everything on the hard disk and let you retrieve that by a partitions and various brokers that you have These are the few important concepts for Kafka if you're getting into Kafka This is how these are the basic things that you need to take care of clusters So cluster is basically set of brokers now you can have one broker Single node single broker you can have single node multiple brokers you can have multi node multiple brokers It totally depends how you want to configure your Kafka cluster, but Kafka lets you do that you even if you have say You are a multi data center environment Kafka lets you replay your data across data centers as well something which is known as Kafka mirroring Topics so topic everybody coming from a message queue background No topic is just a layer which lets you group your messages together now the advantage of Kafka is I have multiple consumers reading the same topic Now each consumer can start from a different point and read to a different point. It's not like Say I have read a red n messages from one consumer So each consumer in that layer would read from n plus one point that is what is known as offsets offsets In Kafka so along with zookeeper you can maintain your offsets wherein you say this topic has been read until this point So it's simple like a tail minus F that you do on a log file Partition is basically you use it for application and you can partition your topics across multiple clusters So it's basically how I how I treat it as rethinking of how logs work So basically in your environment don't treat it as logs treated as a log centric environment wherein your logs will be used by multiple services for example your For example, you're pumping logs now one application is processing for real real-time analysis And the application processing it for probably say a daily analysis you're persisting that to her do so It's the same log, but it's been getting used at a lot of different at a lot of different layers How we can use Kafka with Python there are currently two Two implementations to that one is Kafka Python That's the that's the first module in the community and the other one is samsa. That's again open-sourced By the guys at parsley I'm not sure what version does samsa work on but Kafka Python works on the latest version of Kafka as well I'm not very sure of samsa. I haven't tried that out Kafka plus Tom really makes a near real data pipeline a very robust and a fault tolerant mechanism for you So if you're looking forward towards implementing that you should give it a try both of the systems work independently So you know you are not It's not a mandate to use Kafka with storm or storm with Kafka both of them work pretty much independently So you are free to use any one of them There are a lot of advanced features of storm as well. For example, drpcs topologies transactional topologies tried in topologies now When when you're writing a near near real-time data processing pipeline You have to make sure that each of your message messages get processed Otherwise at the end of it your counts will start to differ right now when we say each of the message gets processed once We tend to forget that it's not exactly once right so there are two implementation to this I will make sure that my messages will get processed at least once and exactly once right so storm Let's you handle both of these things in the basic implementation of storm topology that we saw it will only achieve you And let you achieve the first part exactly once all right because it comes with a reliability API So what reliability API does is let it lets you acknowledge that whether you have processed that couple or not If you've not processed a tuple at any layer during a topology it will replay that particular tuple all right What we mean by exactly once is something that you need to figure out with transactional topologies and tried in topologies That I won't only want to process this tuple it exactly once right so there where there's where transactional topologies come Drpc is supported by multi-lang API, but in the current implementation of the multi-lang API Trident topologies are not supported so probably in future they have that in roadmap But I'm not sure of when they release it for multi-lang APIs These are the resources that I usually follow and these are pretty good. The first one is the official github wrapper for the stream pass library Currently they're working Extensively on that and they have few releases ahead as well But in the current system also it lets you manage your whole storm cluster pretty well For example, you want to ship your logs from various workers and see where your bowls are Fultering you can do that from that you can submit your topologies as we just saw onto your storm cluster as well Then you have Kafka and Python libraries official documentations are pretty good for both the system So it's a it's a must treat for all of you and if you want to understand how a storm parallelism works There is a block by Michael Nell. It's it's one of the best blocks that helps you understand how parallelism works works in storm Just like a topology we can continue the discussion forever But I'll put a stop over here and open up open it up for questions and just before I forget I Forgot to show you the storm UI that comes with it So that's basically a storm UI that helps you keep track of how your cluster is behaving. So I Submitted a topology Okay, so that's Okay, so my topology was Wikipedia underscore trains. That's been running since not one minute. Let me refresh that So it tells you the uptime of your topology how your others other parts are behaving on the cluster. Yeah So it's been up from 50 minutes If you click on that you would see the different spouse and different bowls that you have in this So you have spouts and you have balls So I had one spout and I had two bowls count and split so it gives me how many messages are being emitted and stuff Why there's a difference between the total emitted and Okay, yeah, why there's a difference between emitted and transfer because one of my bowls does not emit messages It's it's a simple web socket push Right. Thank you Yes, it's still the case Yes, so the question is an earlier version of storm to update a topology. We had to restart the topology again It's it's the question. So you still yes, you have to do that There are multiple design challenges around that which does not let you do that for example Now depends what kind of change you're doing for example if you're doing simple parallelism counts change and stuff like that Now it depends how your tasks will then we get distributed, right? So you have to bring a topology down There is there is a feature request in it which is known as swamp I think so that lets you do storm swamp So that will help you take care of this part wherein you define your parallelism on the fly and you just do a storm Storm swarm that will basically rebalance all all your cluster in terms of the new parallelism that you've defined But I was reading an interesting mail chain on the mailing list itself that had other side to it that why is it very difficult to To create systems where you can change them on the fly and then let them behave the same way because your counts your aggregates How your messages have been grouped across different workers really get affected by that? So I don't think that's that's coming in one of the near future releases for that You mentioned the reliability guarantee So the question is around reliability API Supposing the question is we've written a topology in Python and one of the Components start failing. So what happens to the whole data pipeline, right? Now there are two parts to it. First of all, I Can probably go back to the code snippet So this is how you define acknowledgement in failures in your system So for example failure might not might be because of a component failing or failure might be because of a business logic failing Failing right so in any of the case I can explicitly fail my tuple and that will get get then replayed But yes, if one of my workers go down the supervisor will make sure to restart the workers If for example, say I'm using a module that is not available on one of my workers Stream powers will not let me submit that topology itself So this is how that will take care of that in case of failing of each in message individually The reliability API explicitly lets you do that to acknowledge or to fail either of the messages Both on cases of business logic or any other scenarios that you have The question is how we can scale the machines by adding more number of machines and increase the throughput for the same So supervisor is what runs your workers, right? So all you need to do is add more machines and start running supervisor over it It will communicate with the nimbus and then how you will scale your topology Yes, you have to that that's the second one is coming You'll you have to restart but you and it will not take care of the dynamic loads For example, you know that your data load is going to get increased so automatically I would like to scale my storm cluster storm does not come with with that facility But it's very easy in terms to scale because all you have to do is start a new worker worker around it And all your tasks will then get redistributed Yes, so essentially if you do a ps-ef on on your system and grep for jvm's you would see different jvm's running on it So essentially what is happening is each supervisor runs a set of workers So each topology will have set of workers which in term have executors and threads So each one of them will have a different process. That is how you guarantee fault tolerance Sorry across multiple systems. So one of your components fail. You can still the other components will still run for you So that is how it happens inside Inside a topology. So, okay. So the question is how is Twitter storm compared with different frameworks that are available in the market So samsa is one of the products that is open sourced by LinkedIn itself So linking for its near real-time does a lot of samsa then there's one other product open sourced by yahoo Which is s4. I'm not really sure of those products. I've never used it But yes, there has to be some design philosophy That is why there is existence of few different systems that are around There is one that is used by Facebook. I'm not sort of that's open source or not But I've never really given a shot to other one of them because for me The components that Twitter storm list the features that list were pretty strong and the multi-lang API for me is pretty strong as well So those are my preferences for using Apache storm. Yeah, I Can't hear you can Sure, so the question is how does exactly once and at least once processing works So there's something known as the guaranteed processing wherein you will flow all your messages across your topology Once that is done. You are at least processing it once right now Exactly once is what I was talking about in the advanced features of storm where you need to write transactional topologies Which are abstraction abstraction over? Your base basic topologies that lets you achieve exactly once so for at least once all your tuples will flow through the topology That is what storm takes care of by its communication mechanisms So, yes, there are few internals of storm in which it communicates via zero MQ and other other mechanisms I have few architecture diagrams for that. I'll be happy to share that after the talk That basically explain how each message flows through Different workers for example one bolt one or bolt from spout to bolt and what happens when we explicitly fail it and stuff like that So I'll be happy to share that


This is news click and play things of alien forces. We have reassembled after a break of many weeks where I think all of us were looking at other aspects of our, I don't know, lives and careers and other such things. But we're back. We're back, of course, in the midst of a war in Europe, Russia and Ukraine currently engaged in a military conflict that has grown from a limited sort of theater specific conflict into one that has included large parts of the world that is impacting perhaps all of it, including us here in India, whether it's in terms of fuel prices or other essential products. Trade, energy and other such things. Over the course of the weeks of this conflict, this war, we found that increasingly sport has become a topic of conversation or another front on which at least the economic side of this war is being played out. A lot of pressure being exerted by the West, the US, Europe and NATO and their allies, including widespread bands on Russian athletes, not just teams representing Russia, but also individual athletes who happen to come from that country or carry that passport, were born there, perhaps live and train there. To discuss the merits and the merits of, and the politics, of course, of these sporting bands, we have with us senior veteran journalists, today's sports journalists who've covered a vast range of sports, multi-disciplinary sports, motor sports, cricket, football, the Olympic games and other such. Leslie Xavier, Newstik Sports Editor, of course, and Sharda Ugra are both with us. First of all, yes, and I'll start with you Sharda, your understanding of how the politics of these bands or the logic with which they are employed, what does history tell us about how this has happened and have they had the kind of effect that is desired? You know, these bands that are there, they come along at very selective times because the international press is largely English speaking, Western dominated, the press with the money and the media with the money and the influence and the almost like the cashier running the sort of mental outlook of what the world thinks it should be like. So you have these things taking place, these sanctions imposed. I remember in the 80s, it was so hard to get sanctions to be imposed on South Africa until very, very late, you know, so that the voices that were protesting against the apartheid movement came from the non-Western countries, the non-Caucasian countries, if I can use the phrase, and to get that voice. Yes, yes, and to get that voice, it's like you were, you know, shouting into avoid at some point, whereas now you're seeing that it's, you know, it was in what was a practice that was so reprehensible and so abominable, it took so long to budge. And it was sport that actually did kind of make the final push. I don't think that sport, Mr. Poo, that is going to worry about his athletes not competing, it will just kind of get everybody's backs up. So you have to look at it in several ways, you know, there is a war that is being fought and a trauma that the Ukrainians and that are going through, including the athletes that are there among them, those who have decided to take up arms, that's all very valid. But the sanctions, I think it's almost a, dare I say it's a token gesture because it's not like Europe is stopping oil and gas or Britain is stopping oil and gas coming from there. It's not like, you know, there hasn't been sanctions where it hits the easiest people to go after are the athletes, athletes, singers, artists, musicians, you know, they're the simplest, they're the softest touches in many ways. Yeah. Let's see, do you also see it as near tokenism or is it like slightly more even sinister than that, that if you want to sort of participate in and partake of the wealth and the fame that comes with being part of the global sporting circus, then you must also either remain politically absolutely neutral in this case also the demands being made are not just neutrality but outright sort of they are asking for these athletes to outright decry or denounce their own country. And literally choose the side of the opposition. I mean, how is that a realistic expectation to have on individuals? So the realistic expectations are not apart. If you look at, for instance, the band by Wimbledon on Russian and Russian players, I mean, Rubly who had clearly sent out a message against the war, he has been banned. So it's a blanket ban. So what does the athlete have to do now?  I mean, that doesn't work like that.  And it has various layers to it as well. So even when Sharda said about the selective nature of this ban, we have discussed this previously as well that the conflict that is happening in Ukraine is not the only conflict that is happening across the world. In the past 20, 30 years, if you look at it, there have been wars that have been waged, the Gulf wars, the two Gulf wars, the continuous conflict and violations of human rights that happened in Israel, Palestine and Saudi's actions in Yemen. So many other issues in Africa, Syria conflict. So we all know which all nations are part of those conflicts. USA has a prominent role in many of these conflicts and Britain have had its own role. So by that measure, no American athlete or no British athlete or no Israeli athlete should be allowed to compete in any of these global competitions. But then the outrage came out now and it's also a very concerted effort to push. So the, so the politics that's been played by playing behind it goes beyond just the claim that it is to stop Russia and put its avenue for propaganda, that is sport. It has everything to do with the global, the West and their mechanism to push back some kind of an opposition they are facing in their game, in their game to control the world and the politics and the economics that is there. So the, if you ask me what is the sinister goal behind it, what purpose does it, does it serve? It doesn't, obviously it doesn't serve the purpose of finding a resolution to the conflict as such. I mean these athletes. In some ways it's kind of just, just pushing for a further escalation on a different front as well. Instead of like on the one hand you talk of, or you make statements of how sport is meant to unite and bring people together and sport is a platform for peace and all of that. And then on the other hand you go ahead and sort of unilaterally in fact ban people from competing. So on one side you are saying that it is to stop sport being used as a propaganda. But then on the other hand you are using sport to wield the soft power and attack the other party. So it's, and also you are actually affecting the life, livelihood and careers of athletes who has no clear say or role in this conflict. So they, I mean in all the countries if you look at it, at least don't have a say whether the country is going to war or not going to war. And then I had to say like for instance long back during the Vietnam War, Muhammad Ali came out, he was a world champion then and he came out against the war and he spoke out and he lost years from his career. He was incarcerated for that. So that is the, and we are talking about USA who is the vanguard for freedom and individual liberty and all these things. So when that has happened in the US and history has taught us lessons and you are expecting Russian athletes to come out and speak and face whatever the repercussions that come from it. And I'm not just talking about jail or whatever the law that is prevailing in that country. I'm also talking about alienation because you are part of the country and you are part of a society. And then you are standing and speaking against your own countrymen that could alienate these athletes also. And you can't just expect these athletes to do that. And if they do that, come and do that, that's their own personal choice. So we shouldn't be imposing it on them. So that's a larger discussion that we should be having as well. Sharda, both of you actually, what do you make of the stand that Leslie was mentioning the case of this Andre Rublyov and he's winning this Serbia Open but they're not being able to play Wimbledon up next. Despite beating the world champion, he's won three tournaments this year which puts him, I think, makes him in the men's tennis game the most successful guy this year next to Nadal. So obviously someone who was on top of his game in with a shot of winning the competition. These bands also, of course, detract from the very systems that largely these are Western driven professional tours, television events, etc. Wimbledon is called the championships and it's played in London. But the whole world watches it and it's perhaps one of the most widely watched tennis events in the world. Some of the shine of that tournament goes away when so you're hitting yourself as well. So what is the logic actually, I mean is if any, and what do you make of what like bodies like the International Olympic Committee, for example, what their stand has been on? I think the most interesting thing that's happened with this has been the response of the two tours, the men's tour and the women's tour to say that this is unacceptable. And there was a new story that went out. I don't know whether that's been finalized, whether it's happened. They met to decide whether to take points of, say we won't give you points for Wimbledon because it's a tournament in which the merit that brings players into the tournament based on the rankings that we have is what decides the event. And if you don't have ranking points, then it just becomes an exhibition event. And so if that goes through, it'll be a genuine, a real pushback against this kind of, you know, the use of the athlete as a propaganda tool. Like Leslie said, to counter what you're saying is the propaganda tool of the Russian state as such. So that comes through, it'll be a very big step. And I think in a way that it'll also mean, for example, that those two tours have distanced themselves because it's the tennis players who run the tour. They've said, this is the stand we wish to take. I do hope it comes through because the IOC tends to be where we're all but individual IOC members and we're a family and it's run by the countries that are behind it. I feel the Wimbledon band has come about with perhaps a lot of pressure or a lot of conversation from the British Foreign Office because the Sports Minister said those things. You know, that you have to say you are against Putin and whatever.  For example, you have to ask Marcus Rashford what he thinks about the government at the moment. So to my mind, I would like to see what happens with reference to whether the athletes can actually get some of this back.  I mean, the poor athletes, the Russians, they are not allowed to have their countries flag next to their image. The anthem is not played, the country's name is not taken. I said, how much more do you want to punish them?  So like Leslie said, you have nothing to do with them. They haven't been responsible for starting it and they are not going to be responsible for ending it as well.  No, ATB and WTA stands in that regard because they also had removed the flags from the ranking list obviously. But as far as professional sport is concerned, not just tennis, you have boxing, you have UFC. So in all this, you have NFL in fact, Russian players are still playing in the US. So in all these areas, sport takes the business of sport, takes business as well. So you have Russian fighters fighting for the title as well as well. So in that regard, ATB had a clear deal with Wimbledon or any tournament organizers. The entry criteria is ranking point and if you discriminate with the country, that's actually a breach of contract. So they are mulling action. They are not, I mean, yeah, of course, considering the station and the history of the tournament, there wouldn't be outright removal or anything of that sort. But docking off ranking points is a real scenario. It's not been decided yet. So they are discussing that as such. But the larger point is that professional athletes again still have some sort of respect. But it's not the same for Olympic athletes. It's not the same for amateur athletes, the young athletes coming up and all that. So you are basically cutting that route which enables these... You are talking about Russian athletes who are again considered medal prospects when an Olympic comes around, for instance, or any global event as such. So you are basically attacking those athletes and taking away their chances. And not just this current generation of athletes, but also a few generations would be effective as such. So this is a larger attack on sport as such. And sporting bodies don't realise it. In the politics, they lose that plot, lose that idea. Yeah, if anything, on the face of it, this should be exactly the kind of thing that the IOC responds to by saying, hey, this is government interference in the functioning of sport. We will not stand for it. But somehow that's not how it plays out. Interestingly, the NHL, the National Hockey League in the US, which is perhaps the professional sporting league that involves the maximum number of Russians. Ice hockey being, I think, perhaps the biggest sport in Russia, not up there at least. And I think people will remember in the late 80s, 90s. Slava Fetezov actually, who is the current sports minister in Putin's government was among the first Russian ice hockey players to get over the barriers that then existed and be allowed to play professional sport in the US. Now, of course, he's also on the sanctions list. But interestingly enough, the NHL has not taken any decision, nor is it contemplating any kind of ban on Russian players. So when the NHL draft happens later on, Russians will be drafted. And of course, existing players who are playing for various NHL teams will continue to play there. So it's interesting how where there is an impact, a clear impact. And some of these Russian players are the best in their teams, in their league and draw thousands of television and live in the flesh, stadium audiences, they sell shirts, etc., etc. So interesting that the ban doesn't extend to those guys. But lots of interesting things. In an Indian context, I want to ask you guys, and perhaps we can then wind up this little conversation. I had the opportunity to hang out over the last couple of months with a few. And of course, it started out with confusion, lots of questions. Like what should we think about this? Because India also has a very special relationship, historical relationship with Russia. And even though there may be on the streets very little understanding of the conflict between Ukraine and Russia in this case, but there is not the kind of blanket villainization of everything Russian that perhaps exists in western culture, popular culture. Have you guys spoken to any part of Indian sports fraternity, whether it's journalists or athletes? And what is their kind of take on the bans and where we stand vis-a-vis the war? I mean, of course, it's not that we need to be in a position to in any case take sides. I think it should be without any, this thing that like no war is a good war and is anything that any of us support. But what is the sentiment in general in the sporting fraternity here? Generally, I mean, see, again, the voices against war and against the conflicts is prominent here. But as far as the band is concerned, I don't think anybody, I mean, anybody that I have come across, I have spoken to support, support these kind of moves. So because it is a double-edged sword always and these bands doesn't serve anybody's purpose. At the same time, it also brings down, brings the discipline to sport and to the competing nations as well. So that's why someone like a Jokovic who has come out against speaking against the band.  So he feels that even if he wins the tournament, it would, the achievement would be belittled by the fact that it was a depleted tournament. The same thing with, I mean, if you look at India, the Indian football team played Belarus, a friendly match. And that was within the rule because that's allowed as well. You can play in a neutral when you friendly match. And so it was not against that prevalent blanket band by FIFA that would stop Belarus from playing in FIFA tournaments and all that. And they played. So the fact that India was willing to play friendly with Belarus kind of shows where our politics as well as the sporting mechanism stands as far as the band is concerned. We look at, we have always stood out. So as Sharda mentioned, we were in the forefront when pushing for the apartheid band because there was right in it. And I believe Indian football team playing Belarus is also us standing for what is right and ethical as far as sport is concerned. So in that regard, we have always, I mean, India as a country has always stood out ground as far as doing what seems to be the right move. Of course, these things can change because global politics and push and business and all these are intertwined in a big mess. So when push comes to show everything, everybody gets back. But Russia and their relations have always been so on a popular front. If you ask me, it's very difficult to demonize Russia because Russia has always been a ally for India historically speaking. And there have been cultural exchanges that have been understanding like that. So yeah, so that's what the predominant emotion that I have felt speaking to people in the fraternity with sportspersons.  Yeah, I mean I agree pretty much with what Leslie said with whatever conversations is. The idea of sporting bands kind of hits people and they don't like the idea of it for reasons that are not doping, you know, that kind of thing. So you see this doesn't make any sense because there's nobody involved. I think what's also interesting I remembered with reference to Belarus is that there was an anti-athletes protest against the president of Belarus the last time. They went to jail as well. You know, I think if you need your athletes to be inspired by what they see as their power and their responsibility, the athletes of Belarus actually show the world a lot of it by signing this letter. There were thousands, I mean some of them were in jail. So, you know, I think because of the fact that we have the president of having Russians love their children too, that song that Sting had to produce. So we always knew that because we grew up surrounded by this exchange with the Russians and so on. So there is definitely there's a certain anger at the fact that these athletes have been targeted for whatever, for absolutely no excuse at all. Yeah, fair enough. All right. Thank you both for your time this afternoon. For those of you who are listening to us, this is Playthings of Alien Forces. You can catch us and get more details on these and other stories from our website, newsstick.in. Also, we will now be available on Spotify as a podcast or not just Spotify actually on wherever you get your podcast with this Google or Apple or Spotify or wherever it might be. So check us out there, follow us and we will be coming to you from now on hopefully at least once a week with conversations from around the sporting world with a particular focus, of course, on what's happening in India. Yep, this has been a new skin presentation. Thanks very much. Goodbye.


Hello and welcome to the Grand Line Review, your source for everything one piece. And today we have a review of chapter 963, Becoming Samurai. And this is what I've been waiting for. This is why I've been endlessly excited ever since the Odin flashback began. And we're going to be in straight at the end because Whitebeard is here and the final two page composition of this chapter is the sort of thing that I live for. Seeing this clash between Odin and Whitebeard felt quite surreal during the first read through. And part of that is because how powerful I felt Odin was prior to this. Because whatever that feeling was, it has immediately skyrocketed. Although I guess that actually does make some sense because at this point in time, nine years have passed since we last saw Odin in action. And I don't want to get into power levels and crap because that gets really boring really quickly, but it is simply stunning to see someone in what seems to be an equal bout against Whitebeard, even if it's only brief. In fact, the panel of the clash with Odin actually reminds me a lot of when Whitebeard and Shanks collided, as well as more recently when Big Mom and Kido started their battle. It has that same kind of epic vibe about it. And watching Whitebeard's younger crew being blown away by the clash really solidified that. The whole sequence was amazing though. There isn't a single panel on this two page finale that disappoints in any way whatsoever. Even the first one, wherever you can really see is the Moby Dick on its side and Odin's legs, which are conveying a fantastic sense of speed and impending danger that really justifies Whitebeard taking this situation seriously. And then there's the final panel of the chapter with Whitebeard looking completely befuddled after Odin essentially demands to join his crew. And this is a very rare facial expression from Whitebeard. In fact, I can't really remember another time where he has looked in any way uncertain. Even during the entire Paramount War, Whitebeard was like this beacon of confidence and nothing could rattle him, whether it be facing off against an Admiral or literally being stabbed by one of his sons. He still maintained this aura of supreme maturity like he had seen and done everything this world had to offer. Meanwhile, in this flashback, what I'm really loving about this younger incarnation of Whitebeard is that it seems like the world can still surprise and amaze him, and I really can't wait to explore him a bit more in the coming chapters. And we should also mention Whitebeard's younger crew, of course, because wow, what a group of young upstarts. Seeing everyone together at this age actually reminds me quite a bit of seeing all of the straw hats together, especially Marco, who at this age looks like he had the energy and optimism of a loofy like individual. One character who I did not immediately recognize was Vista though, because he doesn't have the glorious, thin moustache and top-back combination, but it became clear eventually thanks to the swords and of course the proudly displayed chest hair. Meanwhile, it looks like most of the other usual suspects are there. In fact, it's probably easier and more interesting to point out which of Whitebeard's commanders we have not seen thus far. So we don't see Nama, the Fishman Commander, Blamenco, Haruta, Speedjiru, Blenheim, Curio, Thatch, and Ace and Iso for obvious reasons, but wow, that's actually a lot more people missing than I thought. What is very interesting though is that some of Whitebeard's ally captains are present here like Epoida and Whitey Bay, which I find fascinating because I really love the idea that they were former crew members who along the journey found their own reasons to leave and strive towards their own personal goals while still retaining a strong bond with Whitebeard. What I also love is that the Moby Dick washed up on the shores of Wano just like the Thousand Sunny, so that it seems almost like a rite of passage for great pirates and crews to make the pilgrimage to Wano and end up in this exact situation. And I know I've said this before and let's be real, I'm going to say it again, but with how closely the Whitebeard pirates are being tied to Wano and Odin specifically, I find it incredibly difficult to believe that their part in this saga ends with Marco saying, no, not coming. These are people who have clearly not only been to Wano but who have developed a bond with the Kozuki clan. If they don't show up to help the Kozuki clan retake their nation, then it seems like gross negligence for a crew that claims to care so deeply about one another as brothers and sisters. And Odin would certainly qualify as one of those brothers. The only excuse I would accept is if they were all pretty much killed by Weevil during the time skip, but that seems to be pretty dark even for post-time skip one piece. But really there is going to have to be a damn fine reason if the remnants of the Whitebeard pirates do not show up in full force at this stage. But Whitebeard stuff aside though, my favorite part of the chapter was certainly everything to do with Nekomomushi and Inuarashi because they're just so damn cute. Watching them set out to see was kind of like getting immersed in a children's storybook with the simplistic art style and the comment showing a clear yet charming lack of knowledge of the greater world. And somehow these two kids actually make it to Wano. Even if they were shipwrecked. And that is incredible considering everything we know about how absurdly difficult it is to get to this island. They got so incredibly lucky even if it didn't seem that way after being captured. But I guess this is another one of those one piece moments where the power of fate stepped in to make sure that Nekomomushi and Inuarashi came into contact with Odin. Kalamatsu as well of course. And speaking of it has been confirmed that our kappa friend is indeed a fisherman as I think practically everybody in the known world expected. The story about him and his mother though was unexpectedly touching. Another jolt of classic fishman tragedy in the world. So I guess it doesn't really matter where in the human realm you go. Even to a completely isolated pocket fishmen still have it hard. What I'd be keen to know and I doubt we'll ever find out but I would love to know the circumstances under which Komomatsu and his mother arrived on the surface. Because as far as I'm aware it's actually super, super rare for fishman civilians to actually leave their realm of the sea. I mean most of the characters we know who did by the pirates or royalty. And I guess I suppose Kami would be an exception to that but I'd still be interested to find out what purpose they had traversing the surface. Next up though, this chapter also had a great repeat of the scene where Yasu rather generously gave Odin's vassals a ton of money after they tried to rob him which we saw in chapter 943. Except that this time around we got to see Dendro and Komomatsu non-silhouetted. And more importantly, this scene takes on a whole new meaning because we aren't just listening to Yasu's words when he says stuff like buy books to gain knowledge. We're actually seeing a montage of the vassals taking Yasu's words to heart and growing his characters as a result. And I think that this is a wonderful scene because it bridges the gap of the Kinamon we met in his Woodland phase and the Kinamon we know today. As well as the rest of the vassals of course but Kinamon is the one who strikes me as having had the biggest transformation. I mean when we met him during Pankazut he was introduced as a super formal samurai dude and it's great to see his journey to that point. Plus one of the best moments of the chapter was when Odin was wondering why they were all acting so strangely being all polite and stuff but most of all this scene was just really touching because it shows what this group of people were willing to do for the sake of Odin. A man who impacted each and every single one of their lives and changed them for the better. This is a show of loyalty in its purest form and I think that stuff like this really helps to inform the modern day events and gives us a sense of why the vassals are willing to fight an emperor of the sea to reclaim Wano in the name of the Kozuki clan. They aren't the only ones experiencing some development though because Orochi was also mentioned a couple of times in this chapter actually. The first of which occurred when Dendro questioned why Odin was lending money to him and they guessed that that implies that Orochi had lost his position working for Yasu and was feeling some pretty desperate times. However, then three years later Orochi is in a better than ever position seemingly working directly for Shogun Tsukiyaki. So there were definitely some questions here. I mean, I guess it's possible that a situation occurred where like Odin felt sorry for Orochi and somehow got him a job with the Shogun but that feels a bit off to me and I wish we had some more conclusive evidence as to what actually happened. For example, did Yasu fire Orochi after suspecting him of stealing the money from the last chapter which would make a lot of sense and I really doubt that Yasu would recommend Orochi to Tsukiyaki but you know what, whatever I guess. It doesn't really matter in the grand scheme of things. It's just a bit jarring to have Orochi jump around to three completely different life situations in the space of two chapters. Something much, much more important than Orochi though which was revealed in this chapter is that the mountain god is alive and I was more thrilled than I think I should have been to see that giant ball alive and well just with some handy stitches. Not only that, but the tiny ball has also grown and was given the honor of carrying Odin in his procession and speaking of that was a pretty glorious aesthetic triumph of this chapter as well. On par with the clash between Odin and Whitebeard in my opinion. Seeing all of these misfits having matured somewhat into an incredibly respectable force gave me the exact same reaction as the citizens of the capital and it's these little things that make it clear that Odin was destined for greatness with the power to unite seemingly anyone with his unintentional charm and charisma and finally, turning to the cover page this week and we have them germ dudes invading Dressorosa and there's not an awful lot to say about these guys. I mean, this is a very typical one piece cover story development where Odin will introduce a minor group of pirates marine civilians or whatever to interact with the primary individual or group in focus. I will say that I do like the design of the female pirate which actually gives off a kind of a radio vibe but I don't know if that's just me associating the word germ with germ off. I also quite like the big guy in the back very unique design. As for the rest though, they're kind of a man especially the individual who I'm assuming is the captain. He looks kind of like a budget version of East Blue Buggy but I'm really hoping that they have some sort of interesting abilities to display. I doubt it though, because this is a cover story and it wouldn't be unusual for something to happen like in the very next installment having all of them captured and tied up. So yeah, probably the weakest part of this cover story for me thus far but still very keen and on board with it. And that pretty much does it for chapter 963. If you enjoyed this video and the content this channel produces in general then please do consider donating to the Grand Line review Patreon because the support of all of your amazing people is what continues to make this channel possible. And if you'd like to see more videos like this but applied to other anime and manga series then please do check out my second channel New World Review for all of your wider needs. And if you'd like to join the fun at any time then please do head over to my Discord server where a wide array of shenanigan retakes place on a daily basis. And finally please do comment with your thoughts on the chapter. This has been the Grand Line Review and I'll see you next time. According to your opinion is one piece the best manga series. All right, this is probably the easiest question I've ever had in my life because yes, absolutely 100% yes.  Probably, but at the same time there is something so incredibly magical about one piece that has managed to hold my attention without fail for well over a decade and has me coming back each and every week for more. No other series has ever even come close to that. And I always end up losing interest when catching up with them and having to switch to weekly reading. So for example, when I started One Piece I had already previously caught up with both Naruto and Bleach which were, you know, super big at the time but I ended up dropping reading them as weekly series because they just didn't give me that zing I wanted as part of a weekly experience. They were much, much better as a bulk experience and felt incredibly lacking when you were just reading one chapter every seven days. But somehow One Piece is just so content rich that you can sustain yourself on 18 pages a week. And of course, when all of those pages are compiled into a gigantic body of work then yeah, you have nothing less than the greatest manga series of all time. Why isn't Whitebeard as popular? Even though he did everything to be the best character. Hmm, so I've never thought about Whitebeard as an unpopular character at all but I have to admit I was quite shocked when I looked up the results of his One Piece popularity polls. The highest position he's ever attained is 26 which I find a bit ridiculous because to me, Whitebeard is one of the most profound figures in the series. Like he wouldn't be in my top five favorite characters but he would probably be in the top 10 and there's no question that he would be in the top 20. But here's the thing about One Piece.  If you had to choose, would you date Nami or Robin? Yes.