So, today's presentation we, it's about Flipboard is a social aggregation mobile app that aims to transform how people discover view and share content by combining the beauty of ease print with the power of social media. This session will give an overview of the data life cycle. Ashish and Rob will discuss the data strategy and architecture of the power of Python and in-memory processing and the role of QBO and Flipboards, Python, SDK and other cloud technologies to quickly and easily assess data, analyze it and feed it into other models. And today's first speaker, we have Ashish. Yes. Thank you. Hello, folks. My name is Ashish Thosu. I'm the CEO and co-founder of a company called QBOL. So, today's talk is going to be divided into two parts. The first part of the talk, which I'm going to be presenting, is going to be talking about generally the big data landscape, some of the technologies and open source that have emerged there. And I will also focus a little bit about when you're running, you know, what are some of the issues that organizations would face when running at a very, very large scale and especially how cloud helps in that. And after that, I'll pass it over to Rob, who will be talking about in-depth of all the goodness around Python and, you know, in-memory processing and stuff at Flipboard. And that would be a very, very strong use case of how all these technologies are used in real-world environments and, you know, in helping there. So, to start with, a little bit about myself, my name, as I mentioned, is Ashish Thosu. I'm the CEO and co-founder of a company called QBOL. Essentially, we offer a cloud-based platform for big data. That's what QBOL does. And before QBOL, I was at Facebook. I was instrumental in building out the data infrastructure at Facebook. I led that team for a number of years. In the process, I also created Apache Hive. That was one of my inventions and my co-founders' inventions. So, having in this environment, you know, when big data was not called big data, from that perspective, I've seen this industry grow and seen what are the advances and changes that have gone in this area. So, just a little fact check. So, why has big data become so important recently? A lot of this has got to do with changes that have happened in the data landscape itself. If you go back to the 90s, a lot of data that was produced was produced through business applications. This is what I would call transactional data. This is essentially data which should be sitting in a database, created as a result of filling out certain forms or transactions. You know, banking transactions is a canonical use case on which the whole of the RDBMS industry was built on. So, a lot of that was transactional data. But as a lot of things started moving online as the web expanded and the web applications expanded and the mobile applications expanded, what we saw is a shift in the nature of the data itself. What has become more important now in the last 15, 20 years or last one and a half decades has been what I call interaction data. This is essentially data which is generated because of entities interacting, whether these could be human beings, these could be machines. And inherently this interaction data is a lot more unstructured, has a lot more volume and a lot more velocity as compared to the previous generation data sets which are mostly transactional data sets. And that essentially has led to, you know, the previous generation systems are found wanting in being able to handle these data sets and being able to provide tools for processing of these types of data sets. And that's really what has led to the evolution of big data. Now, having seen data infrastructures run and operated for this type of data, of course, the most logical thing that people realize is, you know, you need some sort of an infrastructure that scales with commodity nodes because this is really, really large data sets. You really have to think about horizontal scalability, not really vertical scalability. But one thing which has also become important is as these data sets have grown larger and larger, as more data has become available, the number of use cases where data is applied has also mushroomed a lot. So apart from scalability on just the infrastructure side, a modern day data infrastructure also needs scalability on what types of users and what type of use cases it's able to support. It's not just your traditional SQL data warehousing use cases. It is also machine learning use cases. Streaming analytics is becoming more and more important as well, especially with the emergence of the whole IoT industry, whether it's consumer variables or whether it is in the manufacturing sector. And of course, you know, data preparation as well. So all of these use cases have mushroomed. They have become more complex. They have become more advanced. So as a result, a modern day data infrastructure not only does it need to support horizontal scalability to deal with the volumes of data, it also needs to support multiple interfaces and support for multiple different user personas doing different types of analysis. So if you dissect that further, if you go down, you know, if you dissect that data infrastructure for a modern day company further, you would be able to categorize systems in various different categories. And those categories essentially follow a life cycle of the data. Data emerges. It's created in apps. It then moves through infrastructure to collect that data. So that is what is usually called data ingestion. From data ingestion, it then goes into these different systems, some used for ETL analysis, ETL, some used for edoc analysis, some used for machine learning and deep learning by data scientists, and some may be used just for developers for developing certain applications. And finally, the payloads or the summaries arising from these analysis are either used to drive applications or they are used to drive dashboards and visualizations. So consumers for dashboards and visualizations could be your end users who are essentially taking some actions on the basis of those insights. And, you know, applications could be anything. It could be consumer facing. It could be applications driving certain optimizations and so on and so forth. So that is typically how different layers of data infrastructure have to be in place for all of these things to be covered to really have a platform which can subsume all these use cases and make big data very simple. What has also happened is because there was a void of systems being able to support all these different types of, you know, these different categories at scale with big data, a lot of innovation has happened in open source in trying to fill each of these voids. So, you know, in almost every one of these boxes there is some open source technology which is available today and in certain cases multiple open source technologies are available today which allow you to solve that particular box's functionality and when you plug in all these things together you would get a full blown data platform or data infrastructure platform that can solve big data use cases. On collection side, a standard that has emerged quite rapidly in the last few years is Apache Kafka. There are other technologies also available in the cloud for example Pinesis in AWS but these technologies essentially have really made it simple for collecting datasets, large datasets generated from multiple endpoints and putting them in a place where they can then be analyzed. On processing side, streaming analytics has grown multiple, has actually undergrown multiple different, you know, evolutions. Started off with Storm being one of the technologies under the Hadoop umbrella which was used a lot for streaming analysis. Spark has recently emerged as a technology that is solving this use case especially because of its strong roots in-memory processing which is something that you need for streaming analysis. On batch processing, on classical batch processing area, Hadoop has been there for a long time. Hive attacks that use case quite heavily for doing large scale data processing in batch oriented manners. Machine learning again, Spark has emerged as a strong contender there and there are a few other projects like Sphinx which have also emerged and trying to attack that particular use case. They used to be a project called Mahoot under the Hadoop ecosystem which attacked that use case. And on ad hoc analysis, you are seeing also emergence of technologies like Presto and, you know, Cloud has Impala and so on and so forth which are trying to attack those use cases. Net-in-net, what has happened is in the last few, in the last, I would say, seven, eight years, a lot of innovation has happened in the open source stack where now you have a lot of options available to plug in the holes of these, you know, plug in each of these boxes in a comprehensive data infrastructure which you would be trying to put in place for a company or so on and so forth. And what we see, you know, at Cubol essentially we see a lot of these use cases and we see, you know, essentially see a microcosm of how different use cases are using different technologies, what are their relative merits and demerits and so on and so forth and really, you know, we, you know, a modern-day data team essentially needs all of these things in place. It's not just one piece of technology that cuts across different use cases but you really need all of these things in place, all of these technologies in place to serve your analysts, to serve your data scientists, to serve your developers or even to serve your, you know, end-of-line-of-business users. So that is all great. There's a lot of open source technology available. There are templates that have emerged. There are, you know, architectures that have emerged and so on and so forth. But still a lot of companies find it extremely difficult to put together this stack, to put together a data, a big data infrastructure. And many times in many companies you would see a silo of users using big data. It could be a developer team using big data or it could be a data science team using big data but it becomes very difficult for a company to really raise the bar and create a central platform which all of these personas can use for accessing data and for doing data processing at scale. Both in terms of scale of volume but also in terms of the scale of the use cases. And that is where I think Cloud, in my point of view, plays a very, very massive role. Some of the reasons why companies are not able to achieve a true vision of a central data platform that can solve all these problems is because one day the infrastructure, when you're building this infrastructure on-prem, it is inherently static. There is a fixed capacity that you can deal with. And the result of that is always that the central team which is managing that infrastructure tries to lock it down because the capacity is not elastic. Because the cloud is elastic, it gives an on-demand infrastructure. It gives you a lot of flexibility for that central team to open up this infrastructure for multiple different use cases. So Cloud plays a very, very fundamental role there because of its elasticity, because of its flexibility. It also plays a fundamental role in being a cell service platform. A lot of technologies in the cloud are catered towards building things which are cell service and really if a company needs to move to a vision where they want to create a cell service data infrastructure, it plays a very, very central role. Now putting this stack together, so essentially my message here is that in order to really, really be successful with big data, I think Cloud becomes a very, very strong option to build on top of to build your data infrastructure on top of which it becomes a substrate which allows you to open up data to a lot of users and at the same time keep the operational overhead extremely low. However, the way these technologies are built for an on-prem data center versus the cloud are fundamentally very, very different. And I'll just highlight a few differences there to give you a flavor of what it means to make this for the cloud. The very first difference is the storage. A lot has gone into the Hadoop ecosystem to define the de facto storage for big data to be HDFS in case of on-prem clusters. On the cloud, the storage mechanism is completely different. Object stores are the right place and the right technology to store this data because they do a bunch of things. Number one, they make things extremely elastic. So because they are super scalable, you can essentially store data, small data as well as large data and you can adjust them that way. The second thing is by separating the abstraction between compute and storage, they essentially make compute elastic as well. So you don't need infrastructure to be running all the time. You only need it when you need the processing to happen. So object stores fundamentally are a very fundamental building block of the cloud and if big data has to be done properly in the cloud, they should be heavily, heavily leveraged because they give fundamental advantages in terms of operational efficiencies as well as in terms of elasticity that can be exploited there. Along with the object store, the fundamental building block of the cloud is also elastic compute. I think cloud for the first time gives the ability for infrastructure to fit an application. You know, we've been all taught about setting up an infrastructure and then trying to make the applications fit into the infrastructure. You know, I want to optimize this job. I want to change this because my infrastructure cannot do something like that. My machines don't have enough memory or something like that. With elastic compute and with the flexibility that it offers, you can on the fly change the infrastructure to food the application and this becomes extremely handy especially in big data use cases which can be extremely bursty with a wide spectrum of use cases. So you might need extremely high memory machines for machine learning use cases and very low memory machines for ETL work use cases. So that flexibility also is very important when you're trying to do big data on the cloud and it's very fundamentally different from what you get on-prem with your fixed, you know, boxes that have a certain amount of memory profile or a CPU profile or a disk profile and so on so forth. So when you combine these things together an object store and elastic compute you can get to a platform which makes big data cell service which is able to do that in a completely on-demand manner which also feature-proofs any evolution that happens in any of those projects or any evolution in your use cases because of the fundamental flexibility of the cloud. Now one thing that is constantly brought up as a counterpoint is hey, how secure is my data in the cloud? That picture is also very, very rapidly changing. AWS has a lot of product features built around encryption so have the other clouds. There are a lot of product features built around compliance. And fundamentally when I talk about this these are just some of the features that have been implemented on the cloud to address security but fundamentally when I talk about this I always give people an example of whether they think their data is safe in their own data center versus a cloud data center and the analogy to draw here is you store your money in a safe in your home or you put it in a safe in the bank. Some of the best practices around security are undertaken by the cloud providers and as a result I feel that this thing the security perception of the cloud is changing very, very rapidly so it becomes an even stronger option to build and put together a unified data platform which has all these different engines plugged in for various different use cases all provided within a secure environment and I think that is where most modern data teams should aspire to and should run towards in order to make big data successful for the enterprise. So I will pause there I will stop there and pass it over to Rob who will talk about Flipboard and how they are using big data at a much lower detail level this is a much more high level picture but he will talk about lot more details around usage and real world use cases. Thank you. Can everyone hear me? Definitely some really good points. See now that people we are not using transactional data anymore we are using interaction data so if you remember people weren't using phones awhile back and now everyone has a device so we have insane amounts of data going through our platforms and we need big data tools to analyze those large data sets. Okay, sorry. Okay, so I am going to talk about how we use big data at Flipboard and first I am going to give an introduction to Flipboard and tell you what Flipboard is about and what our aim is as far as how we envision the user experience and then I will talk about some of the techniques and some of the products we use as far as a little bit of information about how we use Q-Ball and we have it available to us. So talk a little bit about Flipboard. So Flipboard is a news aggregation or a news recommendation platform. So we think of Flipboard as a place for your interest. What this is is this is the first launch experience in Flipboard so how it works is you open up Flipboard for the first time and you are prompted to this kind of like this questionnaire and ask you what topics you are interested in news, crafting, entrepreneurship. And what we aim to do is take some information using your usage data and the information we get from this questionnaire and pipe really high quality articles to the reader and make it to where they have a really great experience in reading nice long form content and there is another component of Flipboard and that is we work in the area of collection where you can collect things on Flipboard and you can add news articles to what we call magazines so you can create your own magazine and your friends can subscribe to it and they can read the things you are reading and you can have your own point of view exactly as far as what your take is on coffee or surfing or mountain biking and other users can consume that and it is kind of like a social ecosystem around content. So basically what we are trying to do is whenever a user uses the application on their phone or on the web we are taking into account their interactions with particular entities within Flipboard so we will look at a user and let's say this user reads 10 articles we try to take the usage data that they provided to us and then come up with a good strategy to where next time they enter the application they are presented with even better material so basically the idea is that Flipboard learns your preferences as you use it more and more. So right now we have about 80 million active readers plus 80 million and so as far as the data infrastructure or it is part of what my job is I am a data scientist at Flipboard and recommendations of these particular entities like magazines, articles, topics. So there is two different sides of the data pipeline. So one side is the ETL process and working with Hive and getting all the data into our SQL data store which is Amazon Redshift and that serves as our analytics platform and that feeds into a lot of our tools for analyzing A-B tests looking at our particular, let's say our MAR number over the course of the last year and just basically serving as analytics platform and there is a few other recommendations products coming out of the Amazon Redshift data store but most of it actually comes from our recommender index which is pulled from, the data is pulled from S3 and then moved into what we have in this graph database and this graph database produces recommendations so that is another topology or pipeline so okay we have ETL reporting and here are some of the tools is everyone mostly a Python person here or any Java people a few okay and yeah so we use Spark, Spark Streaming actually we are not using this for production ready tools or anything but this is something we are starting to implement and Spark is actually kind of new to Flipboard we haven't been using that long we have been using a Dupenheim for a long time and SciPy and NumPy these are a lot of things I work with as a data scientist they are generally in memory and we do a lot of our prototyping on our in memory you know do a lot of in memory prototyping I could say okay so I was going to talk about people who ask the question like why use Python for data so Python has a lot of really great libraries there is a really great community around Python and it really lends itself well to data analysis because there is a kind of the scientific community uses Python a lot and what's nice is that in my language I feel that an analyst or someone that may be an expert at using Excel or SQL would be able to pick up Python fairly easily and I could show in this example so I'm going to talk about the word count example which is just a map reduce job for counting the words and corpus documents so you can see here we just have a program so we have a mapper and a reducer and then the mapper is reading line by line in a file and it's splitting and then it's adding one it's creating these tuples of a word and a one and then the reducer is combining these together and grouping them and what's nice about this is that you don't actually have to know anything about inheritance or you really don't have even really have to be that a programmer let's say with a degree in computer science or something. You could pick up Python and it's something where you know the inputs and outputs and you know what you want as a result and then the rest kind of comes a little bit easier than using a compiled language so yeah you understand the standard in, standard out you understand looping going line by line and you just need to know how to run this and where to run it from so this is workhound in Java I'm sorry this is kind of small but you can see there's a lot of you have to understand classes the concept of of inheritance and a lot of these other other entities within this language that may not be as easy to understand to an analyst as opposed to a programmer someone who's you know went to school and learned in C++ or Java and I think now most universities require Java as a language compiled language so this is actually kind of something kind of new I'm not sure why they chose this but it's kind of like an industry standard having everyone learn Java so workhound in Java you need to know about objects, classes, inheritance there's a context variable there's a lot of boilerplate code in this example there's a lot of things to import so it just makes it that much larger of a barrier for analysts or people that are new and also there could be other people math people that don't have programming backgrounds that need to learn all of this so we just want them to hit the ground running and be able to start doing data analysis with Python so there are a few advantages to Java obviously it's faster it is more modular I know that YouTube was using Python for a while and now they're kind of moving on to go and using Java because it's packaged a lot easier and easier to send around and like I said more programmers know Java than many other languages because the kind of requirements when you're in school now you have to learn Java okay so this is the part where I'm going to talk about our data stores our pipelines running from S3 and where the sources and destinations are and how they get from point A to point B and all the transformations we make with our data so usage by far is the most important component of Flipboards data because with that usage we don't really have a product and if an hour or two goes by and we don't get usage data that's a serious problem that's like everything has to stop that has to run every hour and this is something that all of our usage pipes into our products so we basically can't have products without our usage data so we have two different pipelines one of them is the redshift pipeline so that's the one I talked about earlier which is our analytics platform and a lot of this data lives in S3 and it's in a JSON serialized format and we do some transformations on it and then we turn it into put it into a column format and then insert that into redshift and then there's a lot of products that we have that pull from redshift the other one is the recommender index so that is kind of a separate data store, a completely separate pipeline and it still reads from the same location it reads from S3 and we have a collector which collects all the data and transforms it and then throws it into Kafka and then the recommender index or the graph database we use our graph database is what houses all of our recommendations and our latent factor models and basically all the tools we have for providing recommendations to users so that server is off of Kafka and then updates in real time so if we have some new interactions from a user it may be an hour or two and then we're able to provide recommendations based on that new knowledge okay so a little bit about a recommender index it's in memory this is the graph database and it's just one giant box it's about I think 250 gigs and we run all of our optimizations because it's very fast and a lot of what we do on a recommender index has to do with clustering and we have a few clobber filtering models running on there and we also have every user's latent factors or every user's information on their usage condensed into these these factor models which provide us with a good means providing recommendations to users and we're talking about Redshift so this is the this is our ETL setup for getting data into Redshift so we use kubel and there's just two steps here you have a kubel configure and a you can create a hive command running this hivecommand.create she's just a better person if you have any questions on that and so this is basically this python ETL setup this is basically what executes whenever we want to run a job on hive and you see here it's just a we have a few attempts that run we have a timer and we have a hivecommand.run and it runs this query that's passed in to this execute I'm sorry this is really small so this is just a general yeah so this is part of the Redshift pipeline so this is just a general execution of this query so whatever query we have and if you want to run it on hive this is what runs it so I'll talk about the process from our data usage, our usage data straight into Amazon Redshift so this is our usage data pipeline it comes in through S3 and these are just rows of online data and then run through a hive job there are some other operations that happen on the data like there's some IP to geolocation user defined functions that get added and once the transformations happen it gets put into kind of like a culinary format and stuff back into S3 and then from S3 the job there's a process that takes it from S3 and then puts it straight into Redshift and then from Redshift that's where you know a lot of our you know a lot of our analytics platforms consume the Redshift data store as well as like ad hoc queries and things okay this is just the process that puts it from S3 to Redshift I'm going to talk about let's see instance management here so previously we had a bunch of Amazon reserve instances on this VPC and we manage them ourselves we actually don't we have about I say three people on the data infrastructure team now and having a fixed size VPC you have to manage everything yourself and there's changing requirements all the time and it just gets to be unmanageable and unwieldy so it's something that it really helps us to have some type of auto scaling and the infrastructure that we use with the cloud it's really nice for us because we spend a lot less money on reserve instances and if we're not using an instance we don't have to pay for it if we're not running a job on it which is really nice and a lot of these jobs we have them on KubeL and as far as configuration, tearing down and bringing up instances KubeL does that on the fly so we don't have to manage that as well so that's also really nice and so let's talk about auto scaling we have a dynamic data infrastructure and so we have all the jobs running on the same cluster if there's let's say two jobs running at one time if there's a new job that comes in the auto scaling deals with the demand of this new job and then kind of comes up with its own strategy to handle running these jobs all at the same time so with running on KubeL KubeL is the instances are hosted by KubeL but owned by us so whenever they whenever we're running a new job and we need more machines KubeL and then D commissions commissions them as the job is finished and let's say there's a lot of money in the past and this is something that we're really happy to have okay so this is one application we use with Spark so we have a follower suggestion recommendation within Flipboard so as you're going down your top stories or your main feet of articles we use we use a recommendation module that pops up and it's basically a user recommendation module or like people you know on Flipboard so this uses a Spark graph so what we're doing in the past is we were computing follower suggestions using Hadoop and this took somewhere around like 40 hours or so to complete and sometimes it wouldn't even complete and this is a lot of the reason why this has is because MapReduce is not in memory so it takes a really long time to get through a clustering algorithm so we've moved everything to Spark and what's nice about Spark is that everything's in memory so these connections from user A to user B they're basically seen as instead of being on different partitions each user is seen as being each connection is seen as being completely separate so each let's say you have a node of a user this node is if it's referenced many many times that's able to sit in memory so that way you can access it easily so this is so we use Spark so this is talking about how long it took for the Hadoop job to run so 40 plus hours of Spark time was 20-30 minutes and it vertices individually and uses caching and holds on to the most visited so you imagine that if there's a user that's very popular on a particular social media platform that user's going to be accessed more than any other user and it's better to have that these nodes decoupled so that way they're not just living on one particular partition you have to revisit over and over again on disk so this makes computation time a lot a lot faster now I'm talking about a user so like social media connection so there are nodes and edges and let's say if I'm following you on social media then I would be a node and there would be a directed edge for me to you okay I'll talk a little bit about Spark streaming and then I'll do a quick demo at the end and the demo is basically just a topic cloud of user affinities so a user has particular affinities to topics and then basically the more affinity a user has to a topic the larger the node will be and I'll show that at the end so this is basically an interest graph visualization so our interest graph pretty much feeds into a lot of our products and each user has a topical affinity vector and we use this topical affinity vector to feed you articles topic recommendations and magazine recommendations as well as it serves our ad platform as well okay so this is a real-time demo it uses Spark streaming so it listens to Kafka and reads events for articles so if a user reads an article we look at what topic the article is in and then we basically the size of the affinity for that user to that topic and so there's just a few steps here this is it creates a streaming context and reads from an input data stream from Kafka and based on the user's time within an article it applies more weight to that article let's say if the user is reading an article for 10 seconds there's a larger weight than if the user goes into the article then jumps out of it so the reason why we chose this is because users that click into an article then jump right out there may be a sign of clickbait or some type of content the user is not really satisfied with so when the user jumps into an article and reads it for a while we assume that the user has a higher affinity to that topic and we apply a larger weight and let's see and I'll do a quick, I'll show you guys this tool so let's see, I can get this to show up here so these are the topics of Flipboard for this particular user you can see that they read more articles about Google, machine learning, psychology and from these sources Huffington Post Business Insider so this is reading from from Kafka and it's giving for each user interaction with an article there is a certain weight applied to a particular article based on a particular topic based on the article's topic so a few other users here have different topical affinities these users' topical affinities are more general yes? oh no, this is something that was developed internally so I think it's just reading off of Redis and so this basically just shows you what the person is interested in and this is a graph for one particular user and you can see here they follow read a lot of space, photography, Apple news, and Facebook articles and it just gives us a good idea exactly what users are interested in and what this user might be inclined to buy if we wanted to show them an ad maybe they're into photography we'd like to show them an ad for a camera and the more we know about users' topical affinities and their interests the better job we can do in providing with articles and content they're interested in so and that concludes my talk do you guys have any questions? so we have so from the devices the devices send to one of our servers and from there we have what we call a flap server which this is the server infrastructure that talks directly with the phones and all of our devices so from there the data is being sent to that service and the service is taking it, putting it into JSON format and throwing it into S3 and then from there we consume it so basically you could say it's not really in real time but it's in it's basically chronologically ordered so whatever we have in the recommender index is all the data and all the usage for a particular group of users up to let's say an hour ago so whenever we get new data it's put into Kafka and then whatever the last hour of usage is it gets put into our recommender index so that way we could users don't generally come every hour or so but we like them to where you have data up to a certain point and then we just keep on feeding more into it, feeding the most recent and generally as a rule of thumb we don't try to analyze like let's say if you're a user of Flipboard for the last four years or something and we don't actually just try to analyze the last four years of your interest and every article you've ever read we just look at the last let's say like a few months or something or six months of interaction one because your interest change over time and two because we analyze less data so we use a sliding window yes you're doing the map reduced still on the with Hive so in a Hadoop Box or in this case you're saying also it's a spark, why is that are you trying to move to Kafka or is there some stuff that Kafka does that I'm not sure of or in addition to we just use it as a let's see we just use it as a queue so basically we just want to read the most recent usage data off of this queue and we want to update our recommender index in all of our stores using that so the most recent usage data is living on there and also we also use SQS we're trying to move over to Kafka but our SQS queue we're throwing article content SQ and it's being also being consumed by the recommender index so recommend index also has our topics so for each article we're able to get a ranking of topics for that article so that way we can have a better idea exactly like what the article is about and so we can also send it to the right people who are interested in reading it so yes more details as to why you're moving from SQS to Kafka well SQS is really expensive mostly yeah I'm not sure for what I understand the reason why we're doing it is because we already have Kafka and if we're going through the trouble of maintaining it and using it then we might as well just use it for one extra purpose instead of having to pay Amazon more money so we probably spend about a million dollars a month on AWS so it's a lot of money we don't have to we don't want to have to pay more for something we already have the means to deal with right now using Kafka yeah I mean we really don't have that many people on the data infrastructure side so it really helps to have a few chosen tools or favorite tools that we want to use I think also at a larger scale when your scale increases you know people who move from say MPP databases towards Hadoop and Hype as your scale increases SQS technologies become less scalable as compared to what Kafka and those things will offer I don't know about their reason but what I have seen is generally as the scale increases you need to start moving towards these technologies which are more catering towards higher volume data and stuff like that and those projects also that also helps so I think it's a combination of both but primarily the places where I have seen moving from some these are all message think of them as message buses so moving from an enterprise message bus to a cloud scale message bus or a web scale message bus which is where Kafka's of the world reside usually happens because of volume good point that's a really really good question so there's a lot of parts there so she wants to know how we use topical affinity to recommend content to users and she also is asking there must be more to that besides just looking at the topical content and then giving recommendations so there's basically two components so if you look at a high level for recommender systems there's usually two types of approaches there is and let's just say in news so there is the topical component so there's content filtering and then there's collaborator filtering so content filtering has to do with the content of the sample let's say that I want to recommend a news article of someone and a news article is about big data and machine learning and Elon Musk so what we do is we could take the article and extract topics from it and create what's called a topic vector for that document and we could use that information in the future whenever we want to generate recommendations so you can imagine that you have a topic vector of size 40,000 and a user's topic vector which represents a topical affinity for all different types of topics is the same size and you can just perform a dot product and there's two vectors and you can come up with a score so you can imagine that you have for a particular user you have this vector and you basically match this vector against the last 50,000 documents or 100,000 documents that came to the pipe recently and then you can just sort based on score so that's just the content part the collaborator filtering part is a little bit more advanced and a little bit more powerful and that it looks at the users that are similar to you based on your usage history and creates kind of amoebas or like clusters of users around the content or around the usage so in order to facilitate this we use an alternate squares algorithm which is an optimization algorithm and it's known as also known as a latent factor model so what a latent factor model is is a model which could take a bunch of user data which is let's say a large I don't want to say matrix but you have let's say a lot of users and then a lot of documents and for each one of those users they have a particular score for each one of those documents so what you can do is create after you've run this optimization you can create a bunch of factor a bunch of factors for each user and then factors for the documents and what these factors represent are the interactions with the user to the content so and you basically same way as cloud filtering as we do with content filtering you match these factor vectors to represent the user in a document and you go through each one of these documents and you come up with a score and then you rank them and we merge both of these models together to come up with our recommender system and the users the pipelining and everything is everything kind of lives on this graph database and that's where all the optimizations happen and where all the factor models are computed does that answer your question? oh so the algorithm I can't tell you exactly which one it is but it's an alternate least squares algorithm alternate least squares? you're welcome yes? so our graph database is internally developed we call it Podex and it was written in C++ by one of our developers we have boost libraries written in Python so basically you can act on this database like kind of like natively in Python it almost feels like but under the hood it's all running C++ it's really fast it's just one giant behemoth box and all the data lives in this box and it's just a large amount of memory on this instance and that's pretty much all we need for doing this yeah well this is actually before Neo4j before a lot of things I think I'm not sure what year it was written in but it was before all these different tools so I guess I could explain like how it would be used I go into I create like a database instance or a database object and create a the object and I could just type db.user and then brackets and just like in like a dictionary grab like a user ID or pass in a string like some like just string user ID and it comes up with all right have this user which is a node in the graph and then I could say okay I'd like to go through all the views of the user and basically any you know their magazines and anything lives within this users part of the graph and it's really useful for analysis and it's a lot more useful for doing certain things like getting the most recent chronologically ordered documents or likes the user had and it's a lot better suited for applications than let's say SQL because SQL is just one giant spreadsheet almost but yeah that was a good question and I don't think he's going to I'm hoping he's going to open source it soon I've been pestering about it but maybe it's something he developed that he just wants to bring whenever he goes to a new job or something or maybe he just doesn't want to have to support it because there's a lot of work with supporting that I think it's probably like 4000 or 5000 no it's not and it's just in memory so it's not like yeah I haven't really I haven't dug through it yet but yeah it works really well works well for our purposes generally that's written in Java and for ETL it's mostly Java well no way no that's I'm sorry that's in Python I can't remember exactly yeah I mean it really depends I guess but like for whatever I showed you there ETL going from S3 that's all in Python so like what types of distributed processing what are we doing we're doing high processing there are a lot of jobs running probably a better question for one of our infrastructure people that's right no I think we are more of a provider and they use our platform to do that how did we discover ourselves I don't know I just met you like an hour ago yeah but I guess I guess there is a pin point and there is a solution to that pin point and essentially that's how we came together so we specialize a lot more on big data on the cloud Flipboard is using the cloud very heavily and that's where the intersection happened so there is a programmatic interface to Kibble and then there is an infrastructure orchestration piece behind the scenes so the thing which cloud allows you to do is to create an infrastructure on the flyer so if you marry the programmatic interface with the orchestration piece you get a very powerful paradigm where everything is created on demand it fits to the purpose of your application what doesn't happen so this is auto scaling it's not just spinning things up and down it's actually spinning things up and down based on the demand and so the algorithms which figure out we probably need more machines or we want to shrink it down and then auto scaling in a stateless application is easy and a stateful application where data is involved so if you are scaling down how to cluster for example you knock off a couple of replicas and then what happens to that data so you have to make sure there is complexity involved around data placement and stuff like that so that is the central part of our orchestration piece and when you are using the cloud so companies like Flipboard and all can write applications on top of this which are doing these higher value things where you are creating these affinity graphs and recommendations and stuff like that so behind that application there is a whole pipeline of things and one part of that is solved by QBOL and another part is solved by the graph database and stuff like that we are programming straight into AWS because lot of configuration management software is general purpose and when you are doing something very specific to big data then you need access to the daemons and the data or the states of the daemons to know those are the inputs that go into the prediction algorithm to figure out how out of scaling has to be done so we use we have kind of built that out cafe so yeah you can use a cafe, a library to use in our networks in what context you mean like in the cloud I don't know I think I built a cafe box with Kuda and I had to build everything myself are you guys looking to go more in that area or we do support integration with art but we have not looked at cafe as such now the platform allows you so you have to draw a box as to what is supportable by the platform and what is not we essentially draw a box around Hadoop, Spark, Hive and Presto those are the four technologies now you can use cafe libraries through or any generic library you can absolutely use that in kibble however that has to fit into the paradigm of one of these infrastructures for example do you want to hook up write a python streaming job in Hadoop or Spark using using cafe or any of these other libraries you can certainly do that but that becomes beyond the scope of the platform itself so any programmatic library that you want to use like mllib a lot of people use mllib with spark you can use that with spark right right so that's programmatic interface that programmatic interface talks to our interface on job submission execution results movement of data and so on so forth that is the interface that we provide below that is the orchestration piece that happens yeah so I guess it depends on what level you enter yeah right that's very focused so like I'm sure you can use cafe for many other things but what I use it for is for a deep learning network and that just basically it looks at an image and classifies it into one of a thousand different categories that is like so niche right now that it's not even useful I mean for us it may be useful for a few other companies like a camera company or like drop cam or something like that but I think maybe soon the adoption will be more and more with deep learning and but I mean still at the level you guys are at now and you could do it but I think does MLLib have any I think MLLib have some deep learning stuff now I'm not sure I don't know if anyone uses that but cafe you could set up with a GPU but you have to do up the management yourself if you're doing it straight up like that I set it up it took me about a day it's a lot and actually they killed my box because I wasn't using it so now I have to set it up again so it happens and you don't what's that I haven't used Docker yet and I had the pleasure I use an AMI so Amazon Reserve Instance to set it all up it makes it a little bit easier but yeah I think we are out of time is it? thank you very much


So my daily driver laptop for a number of years with this Lenovo X250 I think it had it for three or four years somewhere around there and It's a slower i5. Well, it was faster than and it's slower now because well computers are faster now So I would just finally in need of something a little bit faster and I got this 480 And but one of the reasons I went Lenovo's this question comes up is you know How what made you go Lenovo Tom? Don't you hate freedom if you like Lenovo because it's a Chinese company that At one time had an incident with some spyware that was being put on by an American company You're not wrong that happened and I run Linux on it So I don't worry about that as much first thing I do in these is wipe them All right now that we addressed that this is what the X250 looks like after a number of years of use You can't even see the think pad anymore.  I called this IBM when it was IBM before it was Lenovo They always had been known for great keyboards and that hasn't changed So this keyboard everything worked wonderful for all the years which led me to buying this L480 and I've already started wearing on this one. You can see where I do this with the touchpad a lot This is my daily driver. This is the laptop I use all the time is this L480 now But it died and that's one of the things I want to talk about So this is a follow-up after a 45 days of use on it roughly 45 days I bought it ago and it did die about 30 something days in and it's fixed already This is something I want to bring up because service matters and Lenovo is known for having good service Now my X250 never had a problem in all the years I had it and it's maybe been bumped or dropped a couple times. I'm not gonna lie It's it's been roughed up because I take it out on the field all the time And I've taken this one out in the field a few times and it just it was actually here When just wouldn't turn on before a video. I just decided it wouldn't want to boot and the motherboard was dead as a doornail Check the warranty on it even though we bought it from Amazon who I've heard people tell me if you buy it from Amazon You don't get warranty. This was a new laptop on Amazon. I used one not refurbished or anything so that would probably affect the warranty but Punched in the serial number and away we go go go Lenovo It has a three-year next-day on-site warranty and so I went to the Lenovo site Hit the little chat button Put the serial number to them. They're like, yep That sounds like a bad motherboard after I typed in description took only 11 minutes to find to get a hold of someone and they showed up the next day and replaced the motherboard right here at the office They didn't have to go take it anywhere or no one, you know, no big deal So that is the problem I had with it that they fixed right away and the tech did say this is rare and I got to admit for myself because we see a lot of businesses that we've sold Lenovo laptops to and They have been very trouble-free and I Still have confidence in the product in fact that I got three years warranty means if it did happen again I got three more years of incidents before that's a problem and after three or four years It's probably time to retire it anyways, but my overall besides that minor not having it for a day incident is it's great Now I do run Linux on this at running Papa West 1910 on there No issues at all now. I mentioned when I loaded 1904 The touchpad did have a problem the touchpad problem was you had to Activate it essentially and change a command line switch to make it work out of the box fresh install of 1910 does not require you to do that for the touchpad So this is a now out of the box working laptop and the price is really good now the L series is the Budget-minded if you will series, but the extra features I get like I'm not going to use the fingerprint reader I'm not going to Use a lit keyboard. I mean the light keyboard is kind of nice My x2 50 had it and people say you can swap this for a light up keyboard, but I kind of don't care I bet a touch typist for over 30 I think I learned how to touch type back in typewriter class. Yes I'm old enough to have had a typewriter class in high school would have been in 1992 That's when I learned touch typing and where to put your fingers So I don't really think about the keyboard layout and the keyboard layout is not Dramatically different than my x2 50 had So here's the keyboard in the x2 50. There's the key layout and you can swap function and control in the BIOS That is still a feature on the 480 Where we swapped function and control with the keyboard layouts identical So there was no new learning for me to do to know where my hands are even in complete darkness of The laptop so if I'm in the dark I could still type last thing I'll comment in you'll see a few people commenting that the screen is not as Bright now this is a nice IPS display. It's crisp. It's clear. I had no problems with screen the x2 50 is definitely a brighter screen but This one I do leave it max brightness and I'm fine this one would go brighter But I never had it set to full brightness because it always felt too bright So in terms of Ambient watching it in well even in studio lights. I can use this laptop perfectly fine I will admit outdoors it might be a little bit harder to see but I'm not really using my laptop much outside where and it's not Really a problem still a nice IPS display and it's not glossy So even trying to catch you catch a little bit of reflections because the studio lights will reflect on it It's something we find one of the studio lights. It takes a lot of light to get a reflection on it So it's like yeah, it's not matter of fact I want to say that the touchpad has now become glossier for me using it and so the spacebar from slide my fingers on it So it's it's a great finish on it so my overall I still really like the laptop and I Don't want to hide anything about it. The fact that it did die is unnerving when anytime a piece of hardware dies but we've seen this with every major manufacturer where Out of the box you have had a very short period of time in a dive We've seen hard drives die brand new rate arrays that were less than a month old You know you swap them in there. You're like wow one of the drives just decided it doesn't want to work anymore That does happen what it really comes down to is does the company stand behind our product and support it and a three-year next day On-site warranty them showing up and swap in the motherboard out. That's awesome That definitely made me happy about it. So want to throw that out there So those are my thoughts on this laptop up keep using it And I know some people are upset by this because I put stickers on it I'm going to another event and this is gonna travel with me which will accumulate more stickers on it Which would probably bother more people But I don't know. That's how I brand and sticker my laptops as for my x250. This is now Kind of a backup if I needed it, but it's also our shop Laptop that we're now retiring one of the old shop laptops And it's the one the other techs are going to take onsite and wander around with I Leave links to the hardware review and to take in a part of this But other than the fact that the motherboard died. I haven't had any other issues I know it sounds like a really big issue and it kind of is but they replaced it So I don't feel as bad about it. Thanks. And thank you for making it to the end of the video If you like this video, please give it a thumbs up If you'd like to see more content from the channel hit the subscribe button and hit the bell icon if you like YouTube to notify You when new videos come out if you'd like to hire us head over to Lawrence systems calm fill out our contact page And let us know what we can help you with and what projects you'd like us to work together on If you want to carry on the discussion head over to forums dot Lawrence systems calm Or we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos They're accepted right there on our forums, which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you. And once again, thanks for watching and see you next time


It is my privilege to share this great nation, America, with you. Throughout our lives, we cross thresholds. And all thought, obstacles often stand in the way of our goals. We preserve here, as we understand that conquering them will provide great access to personal development, fulfillment and even eventually self-acceleration.  My personal experience of traversing the challenges of the immigration process has opened my eyes to the harsh realities people face, including you, who to try to become your citizens. Becoming an American, citizen comes with responsibility. It means actively participating in the democratic process and guarding our freedom. It also means leading by example and contributing to our society. It is a life-altering experience that takes time, determination and sometimes even tremendous strength. Congratulations again. Thank you.


Cloudy skies may be nice from an artistic point of view, but clouds are an astronomer's nightmare. And while illuminated cities are a nighttime beacon of modern civilization, bright night skies are also a no-go zone for world-class astronomical observations. So, in the pursuit of pristine skies, ESO, the European Southern Observatory, operates its telescopes far beyond Europe in the remote and arid landscape of the Atacama Desert in Chile. This is the ESOcast, cutting-edge science and life behind the scenes of ESO, the European Southern Observatory, exploring the ultimate frontier with our host Dr. J, aka Dr. Joe Liske. A top-class site for astronomical observations must meet several criteria. To begin with, of course, you need a sky that is free of clouds pretty much all year round. But in addition to that, you also need excellent atmospheric conditions, as well as very dry air, with as little water vapor content as possible. And this is exactly the kind of environment that you find in the Atacama Desert in Chile. The Chilean Coast Range Here, the cold offshore Humboldt Current creates a coastal inversion layer of cool air, which prevents rain clouds from developing. Often, a layer of fog is created, which rapidly disperses in the foothills above the desert. A view from the Paranal Observatory towards the Pacific Ocean clearly shows the top of the cloud layer. In addition to the coastal inversion layer, a region of high pressure in the southeastern Pacific Ocean creates circulating winds, forming an anti-cyclone, which helps to keep the climate of the Atacama dry. The Andes lie to the east, acting as a natural barrier for clouds coming from this direction. So all the possible paths for moisture to reach the Atacama Desert are literally blocked. This results in extremely dry air and clear blue skies, ideal conditions for astronomical observations. But we're not done yet without a checklist of ideal observing conditions. In addition to cloudless and dry skies, astronomers need dark sites and unpolluted air in order to make the best observations. In most places, the world at night is far from being a dark place, and the light pollution caused by modern civilisation can easily be spotted. However, light pollution hinders astronomical observations, as it brightens the night sky and makes faint celestial objects undetectable. Only in places that are far from any cities, like some regions in the Atacama Desert, can the night sky pitch black. Furthermore, because chilly cities are relatively far apart, the air in the Atacama Desert is almost completely free of pollutants and is extraordinarily transparent. Astronomical observations are disturbed by the turbulent motions of pockets of air in the atmosphere. Essentially, this turbulence blurs our images of the night sky. In addition, the atmosphere also absorbs and scatters light. In order to minimise these effects, an observatory should be located in an area with a calm atmosphere above it and on top of a high mountain in order to reduce the amount of atmosphere between your telescope and the stars. Once again, the High Altitude Atacama Desert fits this description perfectly. The Atacama Desert offers many sites at high altitude, ranging from extended plateau in the Altiplano Highlands to high mountaintops close to the Pacific Coast. The Chachnantor Plateau, at an altitude of 5,000 meters, offers ideal conditions for observing in the millimeter and submillimeter wavelength domain. This is where ESO, together with its partners, has chosen to construct the Atacama Large Millimeter Sub-millimeter Array, or ALMA for short. At such high altitudes, there is very little water vapor in the air and the disturbing effects of the atmosphere are kept to a minimum. Cerro Paranal is an isolated mountain in the Atacama Desert, only 12 kilometers inland from the Pacific Coast. This is the home of ESO's very large telescope, which makes good use of Cerro Paranal's approximately 320 cloud-free nights each year. Further inland, within the site of Paranal, another mountain has been identified as an ideal place to conduct astronomical observations, Cerro Armazones. This will be the future site of ESO's extremely large telescope, or the EELT for short. The Mars-like landscape of the Atacama Desert is really a wonderful gift of nature. Its unique climate makes it a first-class location for ESO's powerful telescopes, so that night after night, astronomers can observe the crystal-clear skies. This is Dr. Jay, signing off for the ESOcast. Join me again next time for another cosmic adventure.


Hello and welcome back to the Croft box. My name is Brent and today we are on episode 53 of the privateer walkthrough welcome back last time if you haven't checked it out already we Did a whole bunch of stuff we managed to get up to level 70 aka max nautical level That's probably the most exciting thing in the episode and today. We're gonna be talking to catbeard because yes That's where we're going. We were currently in the glasshouse prison. I managed to walk over here. It's not too far away Again, we are skipping dialogue apart from puppet shows because puppet shows are cool Yeah, we've got a bit that we need to get through today Hopefully we can get through it all I would say we can Because the episode next not yeah next episode will be a huge episode where we've got like four dungeons catbeard my man We're gonna see him Yep, okay can do catbeard don't worry. So basically we just need to get catbeard out of here, which is fine You know, I can understand that I Can understand that you know, I really love how on pirate you can just um spam space You don't have to spam a spacebar. You can just press it and it skips all the dialogue. It's so good. So good But yeah, now we need to hit over to the Admorality foyer, which is quite a way quite a ways away But it is very important to get to And once it is got to that's basically the whole plot of Marley bone just in the Admorality foyer and the ward room. So yeah, welcome to Marley bone everybody I think we've only been here for like two episodes now So we haven't actually been here a lot already. It feels like it's been ages And I honestly cannot remember Marley bone at all because I haven't played pirate in forever But um, yeah, let's head over here Hopefully the quest are will not lead me astray because I have absolutely no idea where I'm going I cannot remember anything at all But yes, hello, this is a cool place. We need to go up the stairs from memory It looks so small on the outside, but on the inside it is massive absolutely massive But yeah captain Aubrey. Hi. How how does it go? How goes it? So now we need to find the armada fortress in the Westminster Skyway So I am going to go do that Um off camera just because it's a wee way way and I'll be back and just a second. Okay. I am Back welcome. We have a new ship. It is I just went and bought a new ship from the vendor over in the Docs over back there over yonder It is a level 50 Level 54 Royal Navy galleon obviously Marley bone galleon and Yeah, we have just found this now. We need to fight the ships. I knew we would have to fight stuff So that's actually why I bought a new ship Um Hopefully I'll just do one fight of these because it's not particularly interesting Ship fights as always, but um, hey, we'll do one just because we can so might as well I called it the valiant glory. I don't know why it just kind of stuck You know, I try and make all of the Royal Navy ships sound really like a Good because I actually really like that the origin Probably one of my favorite The Aquila ones are definitely also up there, but I definitely enjoy Royal Navy occasionally. Okay. We need to heal ah Need to heal. Okay. These guys are actually kind of tough It does say they're great, but that is probably because we are max not a cool little and Yeah, okay, let's onboard this do this fight real fast And then I will continue destroying a whole bunch ships Um nice to be sirens attack that we got as well So yeah, I went and got a whole bunch of new ship equipment for it as well. This is really loud noises. Geez Okay, let's go for a big guns because big guns are cool I'm kind of just wanting to do it like right there Just so they have to walk through Which I think is a good idea. We're gonna move up there You can move there and you can go through that actually Monkey King don't do that You can just go there for now Just for now um Yeah, if a kid kid child boom Lots of lots of the fiery stuff easy peasy Yeah, this is actually a really noisy place.  I actually have to turn down the sound effects because it's just boom boom constantly Kind of annoying Okay Let's see how did this this actually went really well That was actually a really good idea to do that because they're taking a decent amount amount of damage I don't know why it went that way. It's strange, but sure weird, but okay And hopefully now we can just go for a crit buff You can do that you can go right there you can do that that should be fine easy peasy does it nice and good Critical hopefully everybody does the crits.  No, okay There we go. There's crit we wanted Double tap onto you nice double tap again easy peasy and another burst fire. Whoo Bonnie popping off. Good job Good job Monkey King's gonna take a tad bit of damage, but it's fine not too big of a worry And there we go Monkey King killed nice gg and can you kill a lot or yes, you can not because you're bad Okay, we understand El Toro is still bad. He has not redeemed himself yet Unlucky, but hey, that's just how it goes Okay Can really kill immobilized attack. Yeah Nope cannot kill that's unfortunate. That's unlucky. Can I kill that is the question? I think I can right? Let's go for artillery That should definitely kill I would say you can just run up there and you can yeah Well, I mean they did I didn't really need to much don't really need to do anything because they did it's just done the Yeah, I thought he would take a bit more, but that's fine easy peasy GG there we go That is that fight done and I will be back once I've defeated two more of these because that's how many I have to do apparently So yee, okay. I am back We collected all of them at wall nation collecting we defeated all of them I should say we got 10 script on the house fight, which isn't too bad But yeah, they're not too difficult definitely would recommend getting a better ship As soon as possible we can now board this fortress and we are going to absolutely destroy it It's what I am guessing Leave ship. Yes, definitely Definitely leaving the ship And hello, hi We just need to fit these guys in a dialogue. That's even better. No dialogue to skip easy peasy Let's do this. Hopefully these guys aren't too bad the yellow so it shouldn't be too difficult What level are they they are level 52 we are level 54 or something 53 nearly 54 See, I'm not too worried about that. We're just gonna do that sounds fun Bonnie Can just go and do that I guess and so can you and Al Toro is going to give us his buff gg easy clap Shouldn't be too bad Yeah Should be nice and easy I would say Immobilized is fine epic hit easy clap boom blade storm epic hit nice another blade storm possibly No, he doesn't have another blade storm. That is not something that monkey King can do unfortunately, but whatever. It's fine It is fine. Don't worry. There's accuracy and dodge means we will be gooder and Hit more and stuff and things. That's how it works overwatch nice What is next attack? Don't you dare?   It's not cool ain't cool Ain't cool overwatch. Yeah, that's right. Don't come near me. Yeah Hit Al Toro, please. Thank you block. Thank you so much Al Toro. You're good relentless. Nope. Okay, that's unfortunate That's really unlucky actually Wow, that was yep, that's awkward. That's a bit awkward Okie-dokie, let's give some more buffage. We will go for a I mean Al Toro apparently needs accuracy. So we'll do some accuracy then just fell throw Yeah, just focus on this guy sounds really good. This musketeers gonna absolutely snipe us, but that's fine I don't really care about him We can go and charge him up later. So yeah, we'll just get Just get most of these other few out because they will be pesky Super hit nice work. Good job monkey King and a blade storm easy clap Hopefully Al Toro can do something. Nope. Okay, just one. That's fine. Whatever. I didn't want you to do anything. Anyway repel borders Yes, thank you. Thank you attack me mega hit. Oh, no, no, don't hit me. That's not cool blade storm No, thank you. At least he doesn't have relentless. That would be bad. That would be really bad Block nice repost. Yes, please. Hello. Good child. Goodbye. Relentless. Yes. Thank you. Good job Next is who we should probably target. Oh mega. Nice. Nice work. Nice work. Really nice work Okay, let's go for the this guy. I guess Yeah, I'll do this Bonnie can do this you can do that and that I don't think they have Do they have Quick adjust double tap and a quick adjust double tap. So that's fine. They don't have quick draw Quick draw is annoying. Don't like quick draw would not recommend Easy peasy nice. It's there relentless nice Why does my relentless always trigger but everybody else's just doesn't it's kind of weird kind of weird attack Yes No crit this time. I'm fortunate relentless. Thank you Another one. Thank you so much. Yay. Well, there we go to down to to go Two to go halfway Like I said, these musketeers just gonna absolutely snipe us, but hey, that's what happens. That's what happens I need a fort out or I would say Millie's put in in the effort here. Nice work. Millie. Good job. Keep it up What we will do is probably Do I via I think I via you you to get this And you go down there. I'll join you in a second El Toro. I'm just gonna via him just so he doesn't take too much damage Because I do actually care about El Toro He is pretty decent when he actually decides to relentless same with monkey king, but some things just never happened Thank you crit as well would be great. Unfortunately, these guys are Agility people so they're probably not gonna crit as much But hey, you know, it's a good a good try good try, you know a good try Okay, go Millie go you can do it. I believe in you Okey-dokey, let's go for a We could do big guns How much does big guns do? 253 we could do big guns like that, I guess you can you two can do that because you're doing well We'll do that actually would it be better if I I'm gonna discard that I'm gonna discard that as well I don't think it has too much of a use. We're just gonna hit. We're just gonna hit get a crit and a relentless. I Would say nope. We didn't even get a relentless unlucky unlucky. We tried we tried. Oh well No relentless either. Yep absorb that absorb that real good. Okay. We do need to heal Bonnie and so I'll probably do that And I'll let these companions all hit at least Miletus is nearly dead, which is good We do need to do a buff again. So what I'll do is We'll do a reinforce You can do that You can do that. Hopefully Bonnie and can kill We'll focus on the other one.  She can double tap is what I'm planning I think that would be the best idea here Double tap nice. There we go. Just what I thought would happen. Good job And then we can just absolutely decimate this guy and it's all done and easy peasy relentless nice We just need a whole another relentless. Yes, and another one. Yes, El Toro. Thank you so much GG easy peasy. I was not expecting that to end quite that quickly. I was expecting at least one more round But hey, I'm not gonna complain Let's get down to the hold next We meet um Is this where we meet Bishop? I think it is we have a puppet show coming up I'm gonna turn up the dialogue because puppet shows are the best the best things in this game of the puppet shows Like actually the best thing is the puppet shows I really love the puppet shows the circle and I just want a puppet show all the time But hey, let's do this. Let's go down. Is this Bishop? I don't think this actually this may not be Bishop. No, this isn't Bishop This isn't Bishop. This is too. This is too Yeah, this is not Bishop yet. I want to have Bishop's puppet show Bishop's puppet shows so cool. I Choose neutralized I do not hold for anybody Nobody can help me gotta love the clicking click click click Okay, let's go for big guns Right off the bat right there. They'll have to walk through that easy peasy Bonnie can Probably just move There I think is probably the best spot you can go there. You can do buff boy easy peasy Yeah, that shouldn't be too difficult shouldn't be too bad As long as we move all the way down here We will have to get the musketeers to move slightly more towards us So, um, yeah, that's kind of what I'm hoping on Because they are quite a way way which is good for us because they're a pain in the bum These are just gonna run right up block. Thank you so much Post epic hit nice. Good job This click is gonna get so annoying like actually Yeah, that if that click just not not cool not cool at all Not cool clicky clicky noise is not cool. Unfortunately. We have to turn it down. Oh god Okay, yeah, I'm not gonna take damage because someone's already walked in that but that's fine You know the more they take damage the better So the least we have to do which is good. Um, okay, let's go for a I mean, I could just go like all the way over there and do that that could be kind of interesting But kind of also don't want to that I'm gonna do that Bonnie can do that And you too can do that easy peasy That sounds like a great plan Kichal and then we go for another one. Oh and a super hit nice How do we get a super hit when we boost when we have a strength bonus and he does as well? I don't know. I don't know how that works, but sure I'm not gonna complain at all No, don't block blocking is rude.  Disappointing. Yes get blocked. That's right That's right. You thought you were good, but you're not you are actually trash unlucky Unlucky block monkey king, please block just just for once, you know, just block actually that would be good Okay, that's unfortunate Boop. Bye. Bye. Sorry. Sorry.  You two can do this I Think it's probably best if they do that Yeah, that's what we do. That's what we do easy peasy. Hopefully we didn't get a any crit or anything, which is fine I don't really mind Monkey King Al Toro get some relentless. Please. That would be good. Nope. Okay. That's also fine. Don't worry about it Bonnie needs to move. She is getting bombarded by everybody cheap shot. Nice work. Good job Okay, Bonnie needs to actually leave leave this place leave this place Bonnie. I might actually heal her It's probably the best idea I would say We do have a VA we do have crit VA is infinite range Do I VA her? I think that's probably the move You can do that you can do that that should be good hopefully Yeah, VA for Bonnie think is the best move here We can always fought the others if needed or heal because we do have endless heels, which is fine No relentless, which is not cool at all. I don't like. Can you block please El Toro? Nope. Okay. That's fine as well Stop hitting El Toro now Stop it. It's not cool. I don't like it.  I'm gonna try this if we mark this guy Will our pit destroy it? That's the question Bonnie's gonna do that You're gonna go over here and so are you and I'm Going to we don't have bladestorm or anything. No, we just have relentless republes Okay, we'll do that. We'll do that down the easy clap. There's probably something that I could have used but hey, that's fine Relentless nice. Okay, that's good We're gonna see if we mark someone if our pit will go there Instead of just going to a random thing it's we have a 50% chance that it did it automatically or by by chance So um, we will see if that actually changes. I don't think it does but it'd be interesting if it did It would be actually pretty good. It would actually have a use if it did, but I don't I don't think it does No, it doesn't okay definitely does not have a have a chance to do that at all Um, okay. Well, let's just focus on this. I could actually heal Yeah, we'll do that you can do that you too can do this hopefully Yeah, this should be fine if it kill nice if it's a good if it kills are always good First fire nice, can we get a crit from you Bonnie? That would be great. Just saying just saying a crit would be dope Stop blocking. It's not cool. How are you? How do you have so much block? Why is El Toro's accuracy so bad? All good questions, and I don't know the answer to them Well Musketeers are always the hardest always the hardest I find them the most frustrating things in the game Always we'll do that and these two can do that. Hopefully this kills. Hopefully fingers crossed epic hit nice This is definitely gonna kill him not the epic hit, but Bonnie and no relentless unfortunately Double tap this should probably hopefully kill the other guy if nobody gets blocked Okay Like I said if nobody get blocked and then El Toro instantly gets blocked, but whatever it's fine It's fine. Not even our pet could do it. No, so sad sad reacts only sad reacts only What am I gonna do to finish off? Let's just do a big guns or whatever it's called done easy peasy Okay, Gigi we did it nice work easy clap boys I think he had actually had quite a bit of armor because that didn't do much Discarded Armada pistols. No, thank you Let's blow up the powder kegs Yep, let's go back to the Admiralty and get our commission then and then I think we might have to call that an episode because Yeah, basically because next episode it's gonna be huge back so soon. Yes. I am the hold you say Don't worry. I have the service may have used for you after all of it all of what you wanted All that you wanted. I did don't worry and this is the Admiralty or the war room. I should say it's so cool. I love the war room It's actually one of my favorite places in the game Admiral cans. Hi a commission Yep Yep, definitely and then we just speak to him and get a new quest sounds fun. Nope. He's not gonna give me a new quest That's fine. That's fine, too. You know, I didn't really want one anyway Well, there we go guns of Marley bone we need to go talk to Gracie next episode We are gonna have a absolute banger of an episode with our huge long quest this quest by the way Has how many stages? 24 different goals we get Gracie, so it's gonna be an absolute popping popping quest Episode and we've also got one two three four instances. So yeah, you know, it's gonna be great when that happens But anyway, thank you for watching room to like and subscribe if you're a subscribe press the we little bells You know exactly when I post another private here walk through and remember to craft outside the box


This study evaluated the factors affecting intraoperative and postoperative complications in retrograde intrarenal surgery. It found that the presence of residual stones was the main risk factor for intraoperative complications, while the presence of residual stones and solitary kidneys were the main risk factors for postoperative complications. Additionally, the stone-free rate was 57.6%, and the overall success rate was 74.8%. This article was authored by Cunit Ozden, Chetan Volcanostikin, Sahin Pali, and others.


So I'm glad that Mark did his presentation just before me because it makes it page the way. And the question is that I don't have to delve so much into. Now, this paper has seven sections. And I'll name the sections as I move from one to the next. And there is an appendix at the end on pages 10 and 11. So we'll look at that in time. So like Mark's paper, our paper, the bills in mind, is a hypothesis on the origin of all Chinese pharyngealization because we don't think that pharyngealization was always there. So as Mark showed, as Mark said, we reconstruct two sets of pharyngealized consonants. As you can see at the top of page 2 in that chart. And it should now be obvious to you that the inventories of pharyngealized and non-pharyngealized consonants are parallel and in our case they are phonemic. So Mark also showed what the reasons are for reconstructing pharyngeal. So I won't go into that. One thing that's worth mentioning is that Norman 1994 regarded pharyngealization not exactly as a characteristic of initial consonants, but as a characteristic of entire syllables. And we think that it is not the case. So we took the liberty of modifying Norman's theory a bit by saying that pharyngealization was a feature of the initial consonant of the main syllables. And the reason we did that is because type A and type B words, rhyme apparently freely in the O's. And if the vowels were pharyngealized, we would not expect them to rhyme. So we pushed pharyngealization left towards the beginning of the words, presumably in the onset. That's why we have these two series. Now as we observe on pages 73 and 74 of our book, this is typologically unusual, as Mark said. I'm very glad that he showed that it's not so unusual as it seems at first hand. But the point is that here we think of typologically unusual features not as impossible characteristics, but as characteristics which, when they arise, do not last long and stable. They are transient. They tend to disintegrate fast. So the evidence we have for reconstructing pharyngeals dates back to really to the end of the old Chinese period or to the beginning of Han times. So the evidence for pharyngealization can be put at that time. And we have no evidence at all on what pharyngealization, sorry, what type A and type B should be at earlier on. Let's now move to section 2, where we outline a hypothetical model of how such a contrast may have arisen. Our proposal is spelled out in table 2 at the top of page 3. Table 2 has two columns. Type A on the left, type B on the right. And it's in three stages. The last stage is the old Chinese stage. The third stage is the old Chinese stage that we reconstruct. And stage 2 and stage 1 are earlier stages. So for type B, it's very simple. There's no distinction between, there's no difference between the three stages. The consonants remain the same all along. So the varying, of course, are the changes. There's no non-pharyngealized consonants we assume were non-pharyngealized at every stage. In type A, however, it's a bit different. The terminal stage where the pharyngeal consonants arise, we take back to an earlier stage in which you did not have a, pharyngealization was represented by an independent segment, a pharyngeal segment. The pharyngeal fricative, which is the ion of Arabic. So clusters, the difference at stage 2, the difference between type A and type B is that type A words begin in clusters of plain consonants plus a pharyngeal fricative. And that, in turn, goes back to an even earlier stage, stage 1, where type A consists of a plain consonant followed by a vowel, followed by the pharyngeal fricative, followed by a second copy of the same vowel. And then a possible code. So basically we suppose that the type stage A forms lose the first vowel, then the cluster is formed and at stage 3 the pharyngeal consonant influences the initial. And they do a sparyngealized consonants which may not last for very long. Now with this reconstruction in mind let us move out of Chinese into Sino-Tibetan and into the branch of Sino-Tibetan known as Cukicin, including the language Lushai. The Cukicin languages do not have a pharyngealization contrast but they have another contrast which bisects the entire inventory, the entire lexicon and that's a vowel length contrast. Some words have long vowels, others have short vowels. And there is a proposal by Stavostin in his dissertation in 1989 who claims that there exists a correlation between Lushai long vowels, words with long vowels and Chinese type A words and Lushai short vowels and all Chinese type B words. When you have cognates between Lushai and all Chinese they tend to conform to these two correlations. Now with the hypothetical model that I have presented in table two in mind let us look at figure one at the bottom of page three. Figure one has two branches and the root at the top the root is the ancestor of Lushai and all Chinese are assumed to be proto-Synotypetan. So the right branch that which goes into all Chinese is basically the same development shown in table two that we just saw. Now how does one go into Lushai very simply by losing the pharyngeal consonant between the two copies of the vowel. So that gives you in fact a contrast in Lushai between words with short vowels and words with long vowels. Very simple. At first sight it may seem a little cumbersome to reconstruct the two things together pharyngealization and vowel length and of course it gets easy to derive both but the point is that similar contrasts exist elsewhere in East Asia specifically in proto-Austronesian and in proto-Austro-Asianic. Now let me first tell you about the Austronesian contrast and that will be in section four in page four. Now Austronesian is interesting in that there is a strong preference for dicelabic words. There are almost no, you cannot reconstruct almost no monosyllabic words to pro-Austronesian unless they are function words. Content words are not, monosyllabic content words are not reconstructed. There are indeed meaning associated monosyllabals in Austronesian they exist and you can reconstruct them but interestingly they never occur as monosyllabals presumably because there is something against monosyllabals in that language. When they occur it is either reduplicated or preceded by a dummy syllable or preceded by phonetic material that cannot be made sense of. Now these, they never occur as simple CVC words. Austronesianists have recognized these for a long time and they call them roots. You can find collections of roots in the words of Blust and John Wolf for instance. For instance, one root, suck, S-E-K, means crumb, crowd, something like that and incidentally it seems connected with Chinese site. Now you cannot reconstruct the form, it's never a free form.  In pro-domaleo-polynesian which is a branch of Austronesian you have hasuck to jam, crumb, crowd sounds like bes-sek, das-sek, das-sek with similar meanings.  Now these forms in which you have a root, these monosyllabic words where you have a root at the end and material that you cannot recognize at the beginning are probably old compounds in which the first part of the compound has become unrecognizable sort of like crayon and cranberry because of phonetic erosion. And this in fact suggests very strongly that pro-Austronesian was stress-final. And so that's why roots occur only at the end of words. Otherwise they might occur at the beginning of words or in the middle of words. Now in fact there are a couple of languages where roots occur by themselves without any added material without reduplications without dummy syllables and not in compounds. Of these two languages one is in Taiwan, it's Boonun and one is in the Philippines, it's Sebuani. And the form under which these roots occur in these two languages is pretty much the same. You have the CVC structure of the root emerges as CVVC with a glottal stop between the two Vs. And that looks a lot like type A. Chinese type A. At least what we reconstruct as ancestral to Chinese type A at a sign of Tibetan level. For instance in Sebuano I've cited roots suck. In Sebuano the equivalent I mean the monosyllabic. Well John Wolf uses a term for these forms CV, glottal stop, VC he calls them stretched. They are stretched monosyllabals. The vowel has been stretched, has been germinated and the glottal stop has been inserted in the middle. Now you can, let's move moving to page 5 and table 3 at the top of page 5. Here you find examples from Boonun. For instance a word root, monosyllabic root, T-A-Q-Earth which occurs with a non-stretched form at the end of dicinibals like you see in the second column like Western, Middle, Polynesian But in Boonun the root occurs all by itself in stretched form T-A-Q-Earth The other examples are like that. You also have evidence of the same alternation in personal pronouns. Free personal pronouns are stretched and bound personal pronouns are not stretched. Now let's move to section 5 bimoracity in Australiatic. In fact, the exactly, pretty much exactly the same system is found in Australiatic as was observed by Norman Zied in 2002. Zied calls it the bimoradic constraint. Zied was the first to point out the role of bimoracity in this kind of behaviour. According to Zied in Andersen exactly the same system can be reconstructed in two proto-Austroasiatic. Table 4 gives you examples from four Munda languages Gutob, Joan, Gorum and Sora. The way these languages deal with roots, with syllabic roots with CVC roots is that Gutob reduplicates the first CV. Joan puts a dummy syllable, a dummy E in front of it. Gorum geminates the vowel puts a vowel stop in the middle and Sora does the same as Gorum but loses the first vowel just like Stage 2 in Figure 1. So as they point out what these languages do is that they have a constraint against monomoradic freeforms. They don't want monomoradic freeforms. They want a freeform to have at least two moray and there are various ways in which they can achieve that result. Now our proposal is that the old Chinese type A, type B distinction, and in fact even the Provo Sino-Tibetan type A, type B distinction as we reconstructed goes back to a language a language earlier than Provo Sino-Tibetan that had the same constraint, that had a bimoradic constraint. How do you go from a language that has the bimoradic constraint to old Chinese? Well you do that by making the disyndables monosyllabic and they become monosyllabic by losing their first consonant. So you are left with monosyndables that have no stretching. And then for the type A words the stretched forms, they evolve as such, as stretched forms and there has to be one change which is you have to need you need to change the the global stop between the two vowels into a pharyngeal consonant. And I have no explanation for that it just has to be stipulated possibly one could speculate that the languages to the north of Chinese whichever an ATR versus RTR contrast played a role there the type A, type B distinction being reinterpreted in terms of ATR, RTR. Now this model implies one thing. It implies that Starostin's proposal that there is a correlation between type A, type B in Chinese and vowel length in Cukicin is correct and we have attempted to test that proposal that hypothesis statistically. First we need to set up a null hypothesis that we want to beat the null hypothesis here is that Starostin is wrong there is nothing special there is no particular correlation there does not exist any positive correlation between Cukicin, long vowel and all Chinese type A on the one hand and no particular correlation between Cukicin, short vowel and type B on the other and like Starostin we do not compare all Chinese type word type with Lushai but with Cukicin the entire group and for this we use Kenneth van Beek's 2009 reconstruction of Cukicin there are various benefits in that first the number of forms is more manageable and second single language irregularities tend to be ironed out so in order to do that we scanned the Cukicin material for Chinese comments of course with some knowledge of from what you call evolution both into Chinese and into Cukicin and we had to exclude certain sets of words for instance we didn't want verbs because verbal morphology in Cukicin often affects the length in ways that we do not understand so we left verbs out and worked only from nominals of which van Beek gives a list in addition we excluded open vowel syllables because there is no length contrast in those in van Beek's reconstruction so we left those out we also left out another types of forms that I leave you to look at the handout to find out of course we also excluded words that have both type N and B in Chinese there are many of those like Ru and Na and many others and the same in Cukicin excluded probable loan words like silver and comparisons requiring large semantic shifts like the one I give you at the bottom of page 7 in total we retained 43 comparisons which are listed in the appendix and from these comparisons we built a table table 5 now from the comparisons we have there are 4 possibilities there are 4 situations Chinese type A Cukicin short Chinese type B Cukicin long and these 4 situations are the 4 cells in table 5 now it is possible using statistics to work out the significance of the deviation from the normal hypothesis a measure of the significance of the deviation is the p-value p-value with this type of data it is possible to compute the p-value out of the 4 figures in table 5 using Fisher's exact test and we set the significance level as 0.05 which is the usual significance level in scientific works anything below 0.05 is deemed significant of course more stringent significance level of 0.01 would be even more stringent but we selected the 0.05 level 0.05 a p-value of 0.05 means that the probability of obtaining the numbers under the null hypothesis is 5% so anything below 5% is deemed significant in this case the p-value is 0.032 etc which means that the probability of obtaining these results under the null hypothesis is about 3.2% which is low so at first sight on present evidence there probably exists a positive correlation between Cooke-Kitchen long-short and Chinese type A type B as the established in proposed of course we're well aware that other scholars could have found other coordinates and then if that is the case then it may be necessary in future to modify our list and to recalculate the p-value however notice that the predominance of the Cooke-Kitchen short versus Chinese type B category is very strong and that probably is going to stay now notice that out of these the 43 forms in table 5 there are 11 that do not match expectations they call these mismatches now what is the explanation for these mismatches the preferred explanation in our opinion is that we are not dealing with exact coordinates we are dealing with reflexes of two words which contain the same root one of them being monosyllabic originally monosyllabic and the other being originally dyssyllabic so that would be the explanation for these mismatches and of course the same explanation goes for the variance in both in Cooke-Kitchen and in Chinese for instance in Chinese root enter noop would go back to a dyssyllable noop while na would go back to a monosyllable noop at a pre-Syllabic level so that is about it and I thank you for your attention