So let's take a poll so that I can get a fair bit of idea here. Why you guys are here?  That's their day job, okay? So others are here just because they're curious. That actually makes a lot of sense, because I'm also along with you guys. My experience mainly has been building distributed applications. I work for a company called Typeship. That's mainly what I've done, played with Scala, Acca, play framework for a little bit. But recently I moved to a Spark team. We have a Spark team where we contribute stuff to Spark streaming. So this presentation is essentially going to be... kind of share my experience for the last six months, and the way I'm seeing this space, big data space is evolving.  I'm going to end up with lots of questions here. And maybe that's a good thing, so that you guys can go back and explore this field a little bit because it's still evolving. But one of the things that I'm realizing, I've been doing programming for a while, is I've been watching this big data space. I'm going to define these terms shortly. What does big data mean and all that stuff? From the fence, looking at this world of Hadoop and how it's evolving and lots of interesting stuff, stops happening, analytics. Now there's a streaming bit of things coming in. And as a programmer realizing the types of problems the business is solving, because most of the technology changes can be driven by... typically gets driven by business requirements, is as a programmer's for the folks we haven't raised the hand yet, it's becoming very, very critical skill to have to at least understand how this space or how to work with this large amount of data. Because if you think about it, data is the kind of problem we solve day in, day out in type of programming we do. What is changing is essentially the sheer amount of data that we have to handle here or the way the data is getting produced. I think it's becoming very, very important and critical skills for any programmer out there. So let's define this term big data. What does that mean to you? Shout it out. We're going to make this very interactive otherwise I'm going to sleep. Please, shout it out. What is it big data means? Data is where? High rated data. Okay, what else? Unorganized. Unstructured, unorganized, okay? You cannot fit in in one machine. You mean in the memory, in the file system, you cannot fix it. Okay, what else? Big data volume. That's true. One of the critical problems of big data is where size is a problem. Okay, size becomes a problem. But actually the way I like to define this, this is a word for storing, managing, and making money out of large data set. That's essentially what it boils down to. People want to make money out of it. What's the fast data? Well, typically when people talk about big data, the things that come to your mind is this big gigantic cluster of server data center where jobs are running forever or takes a while, hours and hours to run. So in that comparison, fast essentially comes down to, well, processing is fast. It's almost happening real-time analysis, online machine learning, again for profit. We are doing something to make out something out of it. This is probably much better definition of that. That was a joke. This is probably much more better definition. That is fast data essentially captures, so we're going to look at some of the examples, the different approaches people are making here. Essentially captures those general set of ideas and sees that because it's a very interesting aspect of fast data is a trade-off, so you do. The previous presentation, there was a discussion about online distribution, a large computation. This is all essentially boils down to some sort of trade-offs. And how can we kind of deliver things timely? That's what's happening. I want to know things right now, how my customer is doing. I put the ad out there, how my ad is performing. Questions like that, they want answers to the question. And processing streaming data. So what we're going to do in the next 40 minutes is to kind of parse this idea and see what is that essentially means. How people are using this kind of solutions. I really like this code. This is one of the fundamental things where the space is coming from. The value doesn't appear automatically out of it. You have to do something for it. Now, big picture idea. We'll step back here and what really happens when you say, okay, I have this streaming of data with a large data set. Yes, I agree that value doesn't come automatically out of it. I want to get value out of it. But essentially, all this thing starts with a problem. Okay? I have a question. I want to find an answer for it. So if you start with the problem, then what happens is that, again, big data space of the way this teams perform are very cost-functional, very, very interesting skills. Okay? Because you'd be working with PhDs. Data scientists out there, engineers out there, data analysts out there, very, very cost-functional things, usually. So first thing essentially happens, okay, we define the problem. Can I express this problem in some sort of specification? Some sort of terms. Once that happens, the usually the first data solution looks like is that we're going to get this data feed into some sort of pipeline. We're going to explore what that looks like, what that means.  Again, somebody's saying that one of the biggest problems with big data is that storage, HDFS kind of addresses and distributed file system, right? And essentially what happens is that this is very, very typical architecture. Okay? Once the process is done, the next thing you do is that immediately dump that to a fast storage, in-memory storage, radius, cache, whatever. So that, so that at the end, someone can take that and consume the result back in. Please ask me questions, okay? Whenever there is any thoughts, comments, all are good. Okay, that's a general idea. This is how they try to solve this problem, this big data problem. Now let's look at some of the interesting examples. Here is one example of how some companies are trying to do detect network intrusion. See if there is a DDoS attack happening in my... Is there an anomaly in my access in the network? How can I figure that out? This thing has to be fast, real-time, right? I cannot wait for something to happen tomorrow. So usually this is a very common pattern. What happens is stream of data coming in and the first thing they try to do is that save that data somewhere as soon as possible. Okay? Then you mind that data, process that data. Then you feed into subsystem where your customers, users, somebody can consume and see what's happening in the network. Let's go more specific. This is the fast data platform in Pinterest. What they're trying to do here is that, again, essentially see how users are interacting with their application. Again, very typical. If you see that this was the... This was very generic mental model, right? This is a more specific one because now we're putting specific tools around there. So first thing usually happens is that those data, in this case, getting dumped into something called Kafka. This is your event log. Okay? As soon as possible, save it somewhere. All right? Then the next part in the chain picks it up. In this case, what Pinterest is doing is that they use some tool called Sensor, which usually saves the data in S3, a large storage. After that happens, the data pipeline then picks it up. Some sort of processing happens on top of it. So that I can come up with some interesting results so that I can answer some questions. Again, it all starts with a question. This is another example.  This is a recommendation engine and Netflix. Netflix? It's what Americans do Friday evening. Watch movies, right? And I'm not sure about Canadians, but... Recommendation engine. Now, who has time to browse to all the movies available, right? So I recommend that Netflix is very much recommended, driven. Your page looks very different than mine. So what that means, and this is a new feature they have added, is when you go to Netflix.com, this panel shows up called Trending Now. This is the movies people watching right now. Now, this is not the raw dump of movies happening. On that trending now, all the movies people are watching, they're doing algorithms on top of it and only showing movies that you might be interested in. That's an interesting problem. That's a value add right there. Again, a very typical pattern. Some sort of data collection happens, okay? Data collection happens. Then they try to do some sort of processing on there. Here they're using tool, again Kafka shows up. Kafka is very, very becoming a standard defactor for event logging. And again, some sort of processing happens and we produce the result back in. Again, I will show you a quote for this. Here is another interesting example, is that we want to predict a breaking news. There's something interesting happening. We should watch out for something. Again, a very interesting problem if you think about it. Again, something that you cannot wait at the end of the day and find an answer for. It is something you need to know right now. Okay, so that you can take some sort of action on it. So this is an example where it's driven by Wiki edits, which is very interesting. If you watch out the real-time updates happening on Wikipedia, there's this crazy amount of data gets added. And another interesting thing happens in Wiki is that when something important happened and interesting event happened, there are a group of people who jumps in Wikipedia and try to update the document out there. So Wiki becomes an interesting indicator that something interesting happened going on. Maybe you should watch out for a particular page. But what this all has to do with functional programming? Why are you even talking about this? I'm already bored for a bunch of people here. Well, I think, and I agree with Dint Wampler here, DIC Data Apps is the killer app for functional programming. Actually, there is no other way to solve the problems that we've been talking about here. I have seen so many companies out there just jumping on Scala for an example just so that they can try to solve this kind of problem. Look at this. I'm hoping. I just hope that you can read out the code here. This is an example of what goes on in a... Oops. What goes on in a box like this, where I'm calling a data pipeline? Where we are doing interesting stuff? Typical code looks like this. It's almost like data flow programming. You take stream of data. You map on it, transform something. You then you flat map. That means probably integrating another stream of data. We are combining. We're essentially joining two stream of data. Then you're performing this transformation one after another. This is your functional programming, isn't it?  Go to S3. Save it to dump it to Cassandra or whatever. So what is streaming? Remember, we've talked about the first data's definition is I need to work with... I don't need to work with streaming data.  I have to find answers as soon as possible. So definition of streaming is a type of data processing engine. It's an engine, data processing engine, designed to work with infinite set of data. That kind of gives an example of that, right? Trending now, amount of movies, people are watching on Netflix. I don't have control over the size. Amount of edits happening in Wikipedia. Amount of people trying to log into your network. Infinite. Just continuously, this event's happening, right? We cannot size them anymore. There are other terms that have been used to explain this scenario, but we're going to stick to streaming, okay? So if that's the case, what are the hurdles? If I've convinced you guys that fast data is the way to go, because one of the fundamental evolution here happening, an interesting change happening here is that it was okay two years back when a Hadoop job took five hours to come up with an answer. What business is obviously, it's coming with the business, it's also evolving. They're trying to say, why wait for five hours if I can find the answer in five minutes?   It's a difficult problem to solve. The patch so far have been called something called Lambda architecture. Has anyone read about Lambda architectures? This is the patch that people use right now to solve this problem. This architecture talks about, hey, yes, our streaming solution is not great, but we still have to do that because my customer is saying I want answer in five minutes, but what we could do instead is mix that with our batch process, the traditional model that we have. And somehow kind of combine those two results and come up with a better answer for you. The downside of this is that this is an interesting idea but terrible hack because it doesn't really work in reality because what happens most of the time is you have to implement same algorithm twice. One for the speed layer, one for the batch layer. What the solution we want to do is I want to make my speed layer so fast enough, a better enough, a feature rich enough so that I don't even have to do a batch layer. I want to, my only data processing pipeline will be the speed layer, nothing batch. The next interesting problem in this space is called, is this latency, right? Not only I have to process the data, I need to come up with an answer as soon as possible. So there are lots of interesting change happening in this space is something called probabilistic data structure. And the most interesting tool I have found in this called BlinkDB, this is one of the Barclay project as well. This is doing queries with approximation, with time bounds and error bounds. We're doing stream processing, we're saying, yes, it's impossible for me to process one terabytes of data and give you an answer in five minutes. Yes, I'm sorry, I cannot do that. Then we come up with an idea of the world. Can you come up with an answer which is 95% correct? I can work with that. This is a very interesting space where we need tools like this, where we can have this time bound guarantees or error bound guarantees. Another aspect of this is that when I'm taking this job and running in the server node, the cluster of nodes out there, I need some sort of cache that can work along across the nodes out there. There are tools like Taqqon which is addressing that problem as well. Machine learning algorithms, something machine learning that's going to run on as the stream of data processing through the pipeline. I need algorithm which can figure that out. Clustering algorithm, anomaly algorithm, whatever algorithm out there. This is probably the most important thing that related to us, the computational model. Okay, that's fine. But if you really want me to write a Spark job or Hadoop job, you better have a good API, good model, because this doesn't work. If we ever written a Hadoop job, probably knows what I'm talking about here. Skip this. We know it's all important. So what is my wish list?  I always try to create this wish list. Why not, right? You can always wish for something. And it turns out there are no tool out there which kind of addresses this problem. But what is emerging out of this space is, what is emerging out of this space is a combination of platform that takes these ideas, and so the one tool cannot solve this problem. We have to kind of combine this together. And those tools are Apache Spark, which is not really close real time, but something that works with mini batches.  And another layer, which is a low latency layer. So there is no tool out there which effectively does par event processing and a batch processing together. So what essentially happens is that people are typically combining low latency streams, the streams that can process with load as quickly as possible with something that can do batching as well. Let's take some examples here. So the Spark here is the replacement of this thing I was talking about here. Spark going to be the replacement of our speed layer. And it has the capabilities out there which can essentially get rid of the batch there we are talking about. That's what we are driving for here. So what happens here? If we want to really solve this kind of problem, let's have a real time analytics problem here. What essentially Spark does is that this is where my data pipeline is going to look like. We're going to have this input stream of data. This could be click stream, people viewing movies, someone logging into the network. All this data comes in and Spark streaming uses a concept of mini-batching. The idea of mini-batching is it's not worse like I am waiting for all the data to be saved before I start my job. It is essentially saying as the data is coming in, break into windows, break into batch of let's say one second or five seconds. It turns out that solution is good enough for the 95% cases out there. For 5% other case, we can talk about the low latency alternative. And then this mini-batches gets into this engine where the actual work happens. This is where we write that pipeline code, the functional code, the Dataflow programming code. And spit out this data, this process data, the results, the outcome that gets saved somewhere else. Let's look at a code right now. I'm going to first run it, and then we will go through the code. This is the example for showing how we can use Wiki edits to kind of predict what interesting events happening. So I'm running something called back-end which essentially hooks into IRC stream of Wikipedia and gets all the real-time events as when the people are editing. If we are lucky, we might discover something very interesting happening right now. Or I'm lucky, I don't know. So I set the back-end. Now I'm actually starting my fast data job. Or I am starting right now. What this is essentially going to do now is is obviously connect to that, and you see the time showing up. This is my window size. This is the mini-batch. So we are showing only top 10 or 15 here. So there's some interesting change happening in Joseph Vaughn. How do you read that? So within the last, I will show you the window size, within the last two seconds, or two minutes, that particular file, that particular URL has been updated twice already. I have seen cases where it gets updated five times, six times as well. This is a continuous running. The way the things are designed, it will never end. I will keep it running, unless you look at the code a little bit here to give you the context of what's going on. So this is implemented using something called Apache Spark. This is another speed layer example. The first thing I'm essentially doing, let's start from here at the end. In line number 40, what I am doing here is that I'm saying the context, some sort of context, the runtime, the engine. I didn't start my context, my job here yet. The reason I'm wrapping it to get or create, because if my job has previously ran and it has cached some data, I want to reuse it again. So it's a fault tolerance mechanism. Now let's say we are starting this job for the first time. When you're starting this job for the first time, line number 17 here, what I'm saying is that we are starting the streaming context and we are saying our interval size is one second. That means what I'm saying is that for every seconds, whatever source is, get the last worth of last second data, pack it in one batch and give it to me. So essentially every second, what is my source here? My source here is a socket-based stream. So what we are essentially saying here is that for every second, worth of data, put it in the batch and give it to me. In line number 18, we're doing a checkpoint directory. Again, it's a fault tolerance mechanism, as when I'm caching data, because this is a state-full-stream operation. Remember what's happening. Every time I encounter an URL, I have to keep a track of it how many times I have seen it. That's why I can tell. Then we're going to put in some sort of limit saying, if I see a URL getting updated within a minute, five times or more, then something interesting is happening with that page. Once that is done, the next bit of code probably will excite all the functional programmers in the room. This is my data flow programming right there. That's my transformation. So it's saying that while the stream is flowing through my system, what you essentially do is that I can have n number of lines in there. Split that into multiple lines. Flat map it. Why am I flat mapping it? Can anyone tell me? Anyone understand this code? Yes, so why flat mapping it? Exactly. We are starting with a collection. There could be n number of things coming in. Now, for each one, I am partitioning to a number of lines. So flat map essentially transforming it and flattening the structure. So each element in a collection could produce another collection. What it's going to do is that instead of producing collection of collections, it's going to flatten the structure. So at the end of this transformation, that flat map, I'm going to end up with a collection of lines. In this case, the line is essentially the URL getting edited and the timestamp with that. Once the line is done, essentially I'm doing a map here. Map is essentially doing a transformation. That means taking, giving a string or element, transform that to a JSON object. And once the JSON is done, then I am doing in the line number 24, essentially taking out specific elements out of it. I know this is a Scala code. I don't expect everybody to understand Scala code in here, but that's a general idea. It's all transformation, all pure functions happening here. And another interesting beauty of Sparky is there is I haven't done any work yet. It's actually lazy. It differs the computation. All this once is done, then I am doing in this line here, line number 26, this is where I'm doing the most interesting bit here. This is my stateful computation. Once I have figured out all the elements that I'm interested on, then I'm saying for each time you are finding an element here that matches this... So I'm saying reduce by key. In this case, my key is the URL. See, if the key is the URL and you have seen that URL, increment the count here. Now, remember, this is a very interesting windowing operation, right? I am going through it. I'm only interested in the data worth of last five minutes. I don't care if a file gets updated 20 times over a day. That's not interesting to me. What happened in the last five minutes window? So as when the data elements are going off the boundary of the window, I'm using an inverse function here to kind of get the reduced account back. As an element enters, I'm incrementing the count. As the window progresses, I'm decrementing the count. Make sense? Because I'm only interested in a specific time frame here. Then I create this window duration here, the size of a window, and then finally I'm creating... Don't show me everything here. I'm only interested in putting two in here. In real world, you probably should put 10 or 20. Only keep the elements that is interesting to me. Has more than two counts. Others just drop them out. Still, again, this is what I'm doing. I just created my process pipeline, a description. Nothing else happened yet. Now, finally, when I perform in Spark wall, this is called action, a side effect, an action. In this case, I'm saying print. This is the one that's going to trigger all the computation happening here. Any question? Show me 20 elements. Stop 20 elements. No, so the question is where I'm saving the data. This reduced by key operation is state-full. It's actually saved in a memory right now. That's why you need to have a checkpoint directory. So Spark has this facility that every time-to-time is going to dump that data. This stream is the abstraction, the type. Wow, that was loud. Abstraction. Hello. This stream is the one that those green boxes here are a this stream. Abstraction over that chunk. One more time, sorry. Yeah, yeah, totally. I'm doing an accumulation here. That's why I'm saying it's a state-full operation. As the elements flying up, I'm actually keeping track of it, what's happening. The track is actually limited by a duration. That's why I have this inverse function in here. I'm not keeping it forever. I'm actually taking things down because I will keep reducing when the inverse will kick in and the filter will make sure I drop the element back out so that my memory doesn't grow gradually. The top edits is my process description. If that makes sense. It's not the data yet. I just defined my pipeline. It's a description. It's how I've encoded. That's my program. You see what I'm saying? That's my program. That's my transformation pipeline. The data that I am saving in memory is the data produced by this reduced function. The counters. The data I'm saving in memory is the one that I'm printing here. It's nice abstraction. It almost has this characteristic. If you think about it, SQL is kind of a subset of a functional programming concept. It's with this old relational algebra and the features they use.  All the transformation we do in SQL also. It's like a map. The big data space, if you're really interested in functional programming, big data space is the place where you can explore your skills because they are very math based and functional based. Very, very much. Not to mention the types of problems you get to solve. Moving on. Show me the code I did, actually. Under the hood, what happens here is when I describe, build this top, it is my process pipeline. That's where I'm building it. Once that is built and I say print, then when this program will get sent to you, my nodes, that, hey, time to run and execute this description. And this model has an interesting abstraction over, you know, now this is a cluster.  To ask are, yes, it can. Yes, based on the resources. I mean, the cluster manager is abstraction. If you're using, for an example, missiles here, yes, it can automatically figure that out for you. Yeah. It's a soft real-time, though. But that's an interesting question because if you look at this here, because that's my next topic, that's a good segue. Now, I have no control over the data that's getting produced. Am I right? This is a hot input stream coming in. People are clicking away in my website. I cannot stop clicking. So data coming in. Now, most of my hard work is happening inside this engine because that's where the top edits or whatever thing that we just displayed there, right? Well, let's say for some reason, so let's say my window size is once again. So every second, no matter what happens, this guy is going to keep producing this box system, this chunks. Essentially, this box streaming is a box generator. They have a class called box generator. So they create these chunks out here. Any idea what will happen if for some reason, let's say I'm taking more than one second to kind of do the analysis. So you're producing data every second, but I am usually fast enough, but something is going wrong. I'm taking more time to process your data that you're piling on. Because remember the amount of data, this is all very interesting, the amount of data I get within each chunk have no control. What spark streaming is doing, give me last one second state. I don't care how much you have it. So sometimes my batches will have 10 rows, sometimes it could be 100,000. One of the biggest problems, that's why I was saying there is no tool out there which kind of addresses this problem. But any guess is what will happen if I slow down. The producer is faster than the consumer. What? All right, I keep queuing. I mean how much I can queue? Yeah, that's one of the biggest problem here with the spark streaming had this issue is you're going to keep crashing because you really don't have control over the producer the data is coming in. So they didn't have a concept of back pressure. Back pressure is a mechanism in each way you control the flow of the data. You control the demand. So typically what happens, there's another new kid in the block it's called reactive stream which is trying to solve this problem. So this also gives you an indication that streaming is a very hot topic these days in a big data space and a way it's evolving. The idea of reactive stream is is there a way I can take that because it's a very, very common problem out there, very, very common, that I'm not keep up with the data is coming in. I want to be real-time. The problem with saving in the file system or doing it later is again going to the batch mode, right? Because has anyone heard the term called code is your liability? Narish and I was discussing about that few days back as well. Your source code is a liability. Anyone agrees with that idea? More code you have, more bugs you can have there, right? And all that, all the things like that. The interesting evolution is happening is that data is becoming a liability now for lots of companies. Yes, we have spent the last five years collecting data, but I have no idea what to do with this, right? So this is also an evolutionary state to fast data. Hey, let's start with the question, what answers we're looking for, and go through this data pipeline and then do the processing. Once we're done the processing data, I don't even care about that. So that's also going again saving the file for something else and data. Going back to this idea of that, I want to do a stream processing, but again have to worry about because I don't have control over the way the data is getting produced. So reactive streams kind of taking that idea and saying, hey, there got to be a way to solve this problem. There's got to be a way where I can communicate my demand, my flow, request, the amount of data I can handle in non-blocking fashion. The way it looks like this, a publisher, it could be any source here. You only send enough data to Subscriber based on his demand. This is how TCP works, by the way. We are taking that idea, borrowing that idea here. We're saying Subscriber should be the guy who should be controlling the flow of data here. Mostak has done a tremendous, there was a morning session in here about reactive streams to kind of talk about this idea here. And it goes beyond this model because you cannot just say, I will do a pool-based streaming system. That means, okay, how about consumer just pull the data? It doesn't work that way. It almost has to be dynamic because some point your producer might be slower and consumer might be faster. So it's a dynamic mix. An ideal streaming solution should have this mix where it can switch between push and pull. Not to mention you can merge data and streams all together. You can skip this. Let's look at another example because this example is interesting to me because not only, oh, yes, okay, you're doing a count on a running stream. What's a big deal? Okay, I'm incrementing a counter. This is not very interesting in a way. But this could be an interesting, right? What's interesting in here is that I am observing my network traffic. Let's look at a problem because that's probably the best way to explain this thing. And yes, I'm saving it some sort of log so that I can mine this data. While I'm doing it, there could be an... I have a specific requirement where I need to process this. Every network request coming in, I need to do some analysis on this. So that per second batching idea will not work. Okay, the requirement is I need to do par event. Okay, eventing system and streaming is very correlated here, par event. Every network, I need to, let's say, let's take a simple example. Every time something logs in, I want to show that in a map here in some dashboard. A login happened from China. Logging happened from North Korea. Logging happened from somewhere. And then we produce this data and not only that we do, we also need to figure out, hey, this network request looks valid. Maybe something weird going on. I want to do a network intrusion detection here. Maybe this request looks odd. How can I do that? Why are the stream of events happening? So there is another evolving space in here is that how can we take this traditional machine learning algorithms which usually work typically you give a set of data, training data, right? Give enough training data to the computers so the model can understand, okay, now I can tell, if you show me a picture of Apple, I can tell it's a fruit, okay, or it's a red color. Then you take that model and you say, okay, here you go, run on the real data and let me give me the detection. But the problem we have here is that first of all, A, we don't even know what the odd request looks like. We cannot, it has to be completely unsupervised here. I cannot even tell you. I don't even know how if somebody doesn't hack on it, how it will happen, I cannot tell you. Because if I can, I will fix that immediately. Second is that I cannot give you enough training data to do that because I don't have the data set anymore. I don't have that data set anymore. You need to automatically evolve and learn. So there is a new space completely happening where people are building online machine learning algorithms, set of algorithms that can learn as the data is processing through it. So in this case, what essentially we could do is that if we have to solve this problem in Spark Streaming, is we have the standard usual suspects in here, check pointing and all that. Then there is a solution called Streaming K-Means. K-Means systems are clustering. And we are starting with some random data set because I don't look like it. The center of this cluster is this. Pick out some random three centers for me. Usually K-Means, the way work is that usually if all the normal HTTP requests, TCP requests happen, they will cluster together. They all look very similar in one group. The one that's left out of that group. And then usually what you can do is that you create your... Oops.  And give me the prediction result. Is there any interesting event happened in the system that looks odd out of this? This is a very, very interesting space where people are trying to solve this kind of problem. Well, I got actually to explore a little bit. I know it was not much polished. Lots of opposite and weird ideas. But any question, any thoughts, comments, share your experiences. That's a great question. So the question is, I will repeat the question. The question is, all right, I'm communicating my demand channel to my publisher. What happens to publisher? What publishers do? I mean it could be a publisher to go to its source and try to reduce it there. Could be a possibility one way. Another common implementation of back pressure could be just drop the packets. Or drop the oldest one you have in the queue. That's also a very common way because for an example in one of the cases we solve this problem where there is a company out there, they want to do real-time traffic analysis. I want to know what's going to happen in Hasu Road right now because I'm living from work. And if you have a data queued up, that gives you a traffic information half an hour back. There's no point. There's SLS in there. So that could be another interest. We didn't prepare this though, but thank you for asking the question. We did contribute a back pressure implementation to SparkStream. It's a PID based though. Yes. I don't, but I will in five minutes. No, I meant 95% of SparkStreaming can solve for other 5% even specific there are other two. I mean, I am saying if I can find an answer in five minutes, why should I wait for five hours? What I am seeing is that it might not be true in very soon.  But what's happening in this space is streaming solutions are getting smarter, better and better and better. So if I, essentially, if I can do that, then I don't have to worry about it. I mean, there was a, there was a great blog post about the whole Lambda architecture, the drawbacks of Lambda architecture and another, there was an alternative proposal called CAP architecture, right? The whole, yeah. But again, this is, we've talked about a little bit of vision here, a wish list. I'll get back to you. Probably it's my jet lag kicking in, but there are others. We should talk, let's look it up. But what's happening is that the machine learning package in Spark is one of the most highly contributed active project right now. Lots of people are contributing there, so it's gonna get better. Yeah. Yeah. Mm-hmm. Yeah. Well, in this case of Spark, the story is a little better. Yeah, maybe we can, we should explore this idea a little bit. I don't have an answer to it, but in Spark, there is a way to share the same code. You don't have to write it twice. There is a method called transform using which you can use the same process pipeline code. Yeah. Well, yeah, that's a trend we're seeing as well. Lots of interest in this streaming field. Well, thank you very much. I hope it was useful. Thanks. Thank you.


Joseph Smith presented a fraudulent document that is not translated the way he said it was. That is a fact in evidence against Mormonism being true. I don't believe that is a fact. You claiming that a fact doesn't make it so. You think that there are Egyptologists who can show that Joseph Smith translated the book of Abraham correctly. Wasn't a word for word translation. How sparse was it? He looked at these things and then he wrote the impressions that he got, an act of revelation. Do you accept the fact that no Egyptologist supports this? I don't speak on behalf of all Egyptologists. Can you show me one? I don't know any. I can show a list of people who object to it. You can't show one who supports it. Does that stand as a fact in evidence against Mormonism?


the tragedy we all know that it was a tragedy but we're talking about an incident in which a plane on radar was observed coming in the direction of a ship in combat and the plane began lowering its altitude and so I think it was an understandable accident to shoot and think that they were under attack from that plane I've ordered an investigation and I'm going to wait till I hear the result of the investigation. I don't go much by what the Iranians say ever. Are you worried about retaliation on their part? You have to think about that knowing who they are. With regard to the Soviets comparing this to the KAL shootdown there was a great difference. Our shot was fired as the result of a radar screen of a plane approaching it at quite a distance. Remember the KAL a group of Soviet fighter planes went up identified the plane for what it was and then proceeded to shoot it down. There's no comparison. Of course I do it a great tragedy. Should stay in the Gulf. Yes. Thank you Mr. President. We want peace in the Gulf but we also want the right of navigation of international water.


what it actually means to live in a place of nature? Yeah, if it was our already part of it. Well, so I'd like to turn to the text here because there's a controversy about what this means. It's not very clear. And even in Cicero, he says, the early Stoics didn't agree, and they have at least three different ways of interpreting what it means. And just think generally about this idea of nature So there is a book called Primitivism in Antiquity by Lovejoy that has an appendix that has 100 different definitions of nature that were used in antiquity, all totally different. Nature is invoked all over the place to explain everything.  But then other people think that homosexuality is contrary to nature. So nature is used on both sides of that divide. Go into the grocery store. Find little stickers that say, natural. What does that mean? I saw one the other day really puzzling. It was on a piece of cheese, and it said, real. And I was like, wow, it's an ontological moment. Here's something that's real. And I was like, cream cheese. So the notion of nature is not immediately obvious what it means. And one thing it can mean is the whole cosmos. Another thing it can mean is human nature, or animal nature, or the nature of living things. And all of those would have different implications, living in accordance with animal nature. I'm an animal, after all. I'm like, woo-hoo, I'm an animal, but I am. And I'm also a kind of plant. And I'm a human. Which nature am I supposed to live in accordance with? Those are all part of my nature. Or am I supposed to live in accordance with the whole cosmos? And how couldn't I? How would you not live in accordance with it? So we have to get further into this question. And Bronson is doing, is actually researching this, is the focus of his paper, what the hell did they mean by living in accordance with nature. And because it's a really intriguing thing that is still around, and we still think we want to live naturally, there's still something about that. There's a kind of normative pull to this idea of let's live naturally. Let's not be artificial. And it could be consistent with the kind of environmental ethics, live in accordance with nature. That's a very attractive notion. We're destroying nature right now. Let's change and live in accordance with it instead. So that's all to say, it's unclear. This is a big philosophical issue. And a slogan, living in accordance with nature, only goes so far. We have to philosophically unpack it. So in section 14 of book four, we are told that even predecessors of Zeno said that the supreme good was to live in accordance with nature. And that the Stoics interpret this formula, so says Cicero, in three ways. So what's the first way? Explain what the first way that they will do is. How do you interpret the first explanation that he gives? Bronson, what are you doing about it? Because we've talked about this already. So it's actually kind of a really interesting question. And I feel like it's easier to work down it. But my 15th kind of start this, that the editor says that this is actually misattributed and also misquoted. So Zeno himself originally just said to live consistently, which of course has nothing to do with it. It doesn't have nature in it at all. And so that could be taken in a lot of different ways. And a lot of research seems to suggest that Zeno meant something maybe even more along the lines of the cynics of just kind of like eschewing almost everything about society and just kind of being a hermit almost. It was Chrysippus who added this like, well, how would you interpret if that's right, living consistently? I can think of two things that could mean. One consistent with something and then I'm wondering what that thing is or consistently meaning I don't contradict myself. I don't do things at variance with my own beliefs or something. Is that what living consistently is supposed to mean? Well, so Chrysippus and the future so it was interpreted to be in terms of consistently with nature. So they added the nature like this to be with nature. Don't test me with that. So you're taking the former. Right, taking the former. So okay, but that's, I just want a gloss or an explanation on the first possible interpretation of living in accordance with nature. So in just in terms of one's knowledge of the knowledge about the fact that nature and thus the cosmos, since that's what they mean by nature is rational and that humans share that rationality and would be acting according to the knowledge of the fact that we have rationality. Okay, so this is like living in accordance with nature means understanding nature, physics, studying physics and then living and then drawing certain conclusions about the nature of reality and living in accordance with that kind of knowledge. Right. So if you have knowledge that climate change is happening then living consistently with that might require you to use consumption or something. Exactly. Okay, so there's one possible answer is that living in accordance with nature means knowing what the nature is, i.e. doing physics and then somehow living in accordance with it. There's problems with that view but that gives us some idea of what it could mean. Okay, what do you want to go on? So the second interpretation is, I really went down with the first interpretation so far in my research. Okay, did anybody else even explain to us what is meant by this second one, which doesn't reiterate the word nature? Yeah. All the way I read it, it seems you should have wisdom with the virtue of wisdom where you know which actions to take and which are appropriate and since living in accordance with nature also means possessing all virtues. If you possess all virtues, if you possess the virtue of wisdom, you know what to do and when to do it and that is the supreme good because you have all the virtues and you are living in accordance with nature. Okay, that's good but it doesn't account for the actual terminology here in section 15. The second interpretation is that it means, quote, to live performing all or most of the intermediate appropriate actions. Okay, so what is an intermediate appropriate action? Let's break that down. Well, I think they're referring to the preferred goods instead of, well, things like wealth when it would be appropriate, it would be and you, it's referring to the notion of preferred goods versus actual good where having wisdom would require you to choose wisely in terms of preferred goods at the correct time. Okay, that's very close. I need to correct one thing you said there but Michael, you wanna come in here? It kind of, I mean, it reminds me of the golden mean or like their style of thing of like a moderation, like by saying the intermediate, like it's like, it kind of reminds me of like gold. Oh, like it's the mean. Yeah. That actually is not being referred to here. Okay, so intermediate appropriate actions does refer to these actions that are neither good nor bad but it doesn't mean, it doesn't have to do with discriminating between the ones that are preferred and dispreferred. That's obvious. We prefer health to disease. We prefer wealth to poverty. We prefer beauty to ugliness. Okay. But sometimes we need to select poverty, disease and ugliness other times the opposites in order to engage in virtuous activity. And so virtuous activity isn't a matter of going, I wanna be wise, I wanna be self-controlled so I'll just do that.  Or somebody puts heroin in front of you and you decide not to do it in that case or do it if that's a good thing to do in that because that's the right thing to do in that case. But so notice that we don't just immediately choose the end. We don't just choose virtue, wouldn't that be nice? We just said, yeah, I'll just choose to be a courageous person. No, it means choosing to hold your line in battle when that is the right thing to do. Sometimes the right thing to do is run, okay? If it's you versus 10,000 Spartans then it would be foolhardy, not courageous to stand the line in battle there. But if it's a matter of protecting your homeland and you can do it and so forth then choosing those intermediate actions is the right thing. So virtue comes down to not just choosing to be virtuous. In fact, that we don't do at all. That's just a matter of what our value system is. The real way to be virtuous is to choose correctly the intermediate things. And so that does require not confusing intermediate things with ends. Like not thinking that health is intrinsically a good thing. But, and of course we need to discriminate between preferred and disperferred indifference. But what it really means is choosing disperferred indifference when that is the virtuous thing to do, okay? So that is a very different sense, it seems, of living in accordance with nature than living in accordance with my knowledge of the natural order as a result of studying physics and doing science. Okay, much more immediately ethically focused idea there. Okay, and so just to give Cicero his due, what's the third one? So somebody explain what he says the third interpretation is. Michael? So he says living in enjoyment of all are the most important things that are in accordance with nature. But he said that to do this is only available to the widest. Where does he say that? Like a little bit further down since however the spring good got set up in this third interpretation and like based upon it is only available to the widest since virtue is a part of it. Okay, and this is where, okay so first of all let's make sure we understand what it means. What does it mean to live in enjoyment of all or the most important things that are in accordance with nature? And how is that different from live performing all or most of the intermediate appropriate actions? Isn't this one, you have to actually possess those things? Okay, you have to actually possess, it looks like this one has, that you have to actually possess the virtues and only the widest do that. Now, in the footnotes, which Bronson told us all to read and we certainly should read when we come into confusing things like this, we're told that this really isn't fair. It looks like Cicero's making up a thing that the Stokes really shouldn't accept. So what is the problem, you think, with this idea that you need to actually enjoy and succeed in accomplishing the things in order to live in accordance with nature? Well, didn't, I don't think they ever actually claimed that anyone was at stage, right? It said that everyone was working towards it but did they ever actually claim anyone to have reached stage with? Well, they have some examples, you know, Socrates, Diogenes, those guys were probably stages. Maybe not, but probably were. Yeah, they're pretty rare. In fact, it might, there might not be anyone. So that's true, but what does that have to do with the problem with this interpretation? There are too many things that aren't in your control which would determine whether you become that. Exactly, and that says he goes on to say that this depends on things that aren't in our own power. Whether I actually, whether it consequentially I end up achieving those things is not in my power. And so we don't wanna make living in accordance with nature, which is the goal, something that is not in my power. First of all, it doesn't recognize the nature of the kind of thing I am, which is a thing that only has control over my own reasoning and in my value system. But second of all, it totally goes against the stoic idea. Stoics are what we call anti-consequentialists. Very strange term, anti-consequentialism. That means they consider the consequences irrelevant. What matters is the state of mind of the person deciding. And so they compare this to an archer. You wanna have the skill of an archer, okay? And you wanna pluck the bow as skillfully as possible. But if somebody moves the target or a gust of wind comes along and blows it away so that you don't exactly hit the bullseye, that doesn't matter. You shouldn't cheat in order to make sure it actually does hit it or something. What matters is the intentions you have behind it. Consequences are totally irrelevant. So they can't really accept this view that you have to actually have the consequence of achieving all of these things. And additionally, there's the fact that it's not clear that anyone has ever achieved these things. It's still the only worthwhile thing to do. But, and there's a couple of examples of people that copy did. As I said, Zeno himself, the head of the school, Socrates, looks like Socrates chose all the appropriate intermediate things. He basically was wise, was courage, did have self-control. So there are people we can point to. Confucius is someone like this. The Buddha might be someone like this. So there are these people and they're doing it right. And we all should be trying to do it right and don't give up and think, well, I'm not gonna be able to become a moral stage so I may as well just pursue wealth or something. That's worthless. And that actually leads, that'll entrap you in vice. At least this other thing gives you a possibility of happiness. Okay, so there's a problem with that third interpretation, but it's a deep problem because it has to do with whether the results and outcomes really matter. And they take a hard line on this. They don't matter. Okay, so the Stoics have easy answers to questions that, for example, utilitarians like to pose.  Would that be a good thing to do? No, because it doesn't matter if you save other people's, five other people's lives. What matters is whether you would do an unjust thing there. Okay, and again, consequences totally irrelevant. What matters is the state of mind and the value judgments made about the things. You don't have control over consequences. So we can't be responsible for them. Now, so those are the views that Cicero gives and not all of them, there might even be something to the third one. I'm not sure if the footnote is right that this is completely anti-Stoic. It could have been a strain of Stoicism or something like that. But those are three different accounts of living in accordance with nature, but more research on it is needed. And there are other questions that remain. If you think that it means living in accordance with human nature, what do you mean by human nature? Apparently they mean reason. Reason is, after all, what makes us humans, what differentiates us from other animals. We are animals, what kind of animals, animals that use language and animals that reason. So I need to live in accordance with reason. But what reason tells me is that certain things are good, certain things are bad, and certain things are indifferent. So I need to live in accordance with those. Well, and so that means selecting the right things that are indifferent so as to bring about the goods and avoid the bad things. And in order to gain that knowledge of what's good and bad, one must understand the physical cosmos and the fact that it is rationally ordered and that reasonable explanations of the causes of things happening in it are possible. Because you can sort of tie in these different elements of the definition together.