tion. It was used for the power station for Pennsylvania Railroad. It had been there since 1930. And it was not used. I mean, the electric, the railroad has not been generating their own electricity for a long time, okay? That's been a long time. But smoke stacks are still there. And they got rid of smoke stack about 18 months ago and I'm coming down the train and like the smoke stack's gone. I'm like, where's the smoke stack? Right? A lot of people don't like change, even though maybe the smoke stack looks hideous when it started. After like seven, eight decades of smoke stack, everyone's expected that smoke stack to be there. When it's not there, everyone is like, something happened. Okay? So don't expect your family to be just rushing to home automation because you think it's important. Reliable operation. You know, people just don't, that's their home, that's their castle. They do not want unreliability. That switch on the wall works every time, folks. If you can't match that, most people are interested. Or let's face it, they don't want complexity. Device longevity, again, if you put something in your home, particularly it's hardwired, is it gonna last for the life of your home?  You gotta think of that. Maintenance, how much are you gonna be able to change this thing? What's the cost? Do you have to have a server running all the time? Like I have a server running all the time, my home, that does a bazillion things. So having automation added to it's not a big deal. But if you don't have a server running all the time, okay, how are you gonna do this? You're gonna make a little, you know, one of those, not Arduino, but the other one, though. Raspberry Pi, yeah, you could. You gotta make sure you get the serial port on there and a whole bunch of other things. You could probably do it. There's not a whole lot of power involved. But again, you have to think of that. And also security privacy. This item here at the bottom, again, from a week ago, actually I made a mistake. I'll have to fix that. The URL here and the URL here are the same, so something's wrong. Anyway, the reason that, you can just search for these titles. You don't even need my thing. Nest thermostat bug leaves users cold. Anybody ever see this article? Yeah, okay. It stopped where they did a firmware upgrade and all of a sudden, some users experienced the problem. I think some means all, I have a feeling, okay. Because if they're not given numbers, then it usually means all. Everybody's got that upgrade and they're worried about houses who maybe are not occupied or vacation residences now. They have no heat anymore. They got to drive 450 miles to figure out what's going on. Nine-step process to reboot it and get it back online. That's scary. People don't like that. So you have to offer something that makes it worth these changes and these disadvantages. So let's get to the beat. Let's talk about actual applications, okay. And again, I'm gonna run through 12 of them. But again, years are gonna be completely different. First one, telephone interface. I told you about the idea that the telephone effectively is a network, okay. So what I do, this is unrelated to X10 at all. I just take a modem. This is a U.S. robotics modem. Believe it or not, you can still buy these things, okay. And as you can see, it's got a serial port. Here's where the telephone call comes in. Here's where it goes out to the computer. And we have a little loop that's always looking to see activity coming. And this is the activity. This is what the X10, this is what, not X10. This is what caller ID looks like in the raw to a modem. You basically turn on caller ID to the modem. You tell the modem, I want you to listen to caller ID. And all of a sudden, you start to get these information. The date, the time, the telephone number, and the name. You can then take the telephone number. You can look it up in your address book. And bingo, you get Christine Momjin from Bruce's cell phone. And you have my telephone number right there, okay. Which happens to be on my website, so it's not secret at all. Okay. If you call it, it'll buzz in my pocket right now. So the point is that, yeah, I've got 20 people calling me. The point is that I've done a couple things here. First, I've gotten information from a network. Second, I've added smarts to it by looking up the telephone number to see if it's in my address book. And if it's in my address book, this description is gonna be better. My description in my address book is gonna be better than what I'm getting from caller ID. And it actually comes up with this beautiful message. It shows up on everyone's terminal, and it shows up on the slideshow, and it shows up on a laptop in the kitchen with X message. Just the whole screen becomes telephone call from whatever. And everyone loves it. This is my daughter's idea, the X10. The idea of coming up on the screen, because a lot of times when we eat dinner at lunch, somebody will call and we'll be like, who is it? And some of them have like, it'll talk to you, but that doesn't help because it doesn't check my address book. So everyone will be eating the phone call ring and will look over and say, oh, it's X person and right away we know exactly who it is. We know if they're calling from your cell phone or from home and we can call them back or whatever we want. In fact, I even have the system set up so I can dial out from my address book. So I never dialed the phone to call anybody. I just say rollo in the person's name, and I say dial, call the house, call their cell phone, call their work, and you just pick up the phone and it's dialing for you because the modem go happy to dial for you, right? You just need to tell it to do it. And also I have a system where if I call the house, it actually makes a little ding dong sound. And it makes a ding dong sound so everyone will know this is not an ordinary call, this is your husband calling, or actually any family member it'll ding dong. To distinguish it from an ordinary call from somebody who isn't in the family, which you're gonna answer but not maybe as quickly. So it kind of is an introduction to life. So it's a combination of the caller ID on the modem, the address book, and X10 to do the chime, and X message to send the message on the screen.  Like you see the terminal saying Bruce is calling, I'm like, yeah, I know I'm calling, I'm here. So it's kind of funny. So again, you can also dial out, and again it's happy to do that as well, automated, okay. This is how it works, the caller ID comes in, it checks the Rolodex and then sends a chime and sends a broadcast message and logs it as well. I have all the calls that come into the house for the last like 12 years I can pull them up. Outgoing calls again, I can force the system to dial directly from the address book. Second home automation would be the first floor, so this is basically a list of all the lights we have, all the wireless remotes, the coffee maker, I'll talk about in a minute, the computer. As you can see, this is pretty target heavy, target rich environment. This is the second floor again, has lights, cool pump is on the outside, obviously, different remotes in different places. So again, this didn't start at once, it kind of grew over time, you've seen pictures of these. But again, as these need became somebody, family member will come to you and say, I would like to have a remote that does X, I would like to have a switch that does Y, I would like this light to go on in this circumstance and so forth, and then you work with that individual to add it. And this is sort of a list of some of the devices that you have, so X10, you have a config file, you name the devices, and you can say X10 off on the family room and all the lights go on. I have a button that turns off all the lights if we're gonna watch a movie, these type of things. This is what, yeah, X10 on cache, turn the couch light on, turn these on, this is the one where if you're gonna watch a video, it turns off all the lights in the area of the video. Again, the idea of using con to do stuff, we talked about that already. This is, again, that same screen where we talked about different things happening at different times. Dawn dusk activity, what I do here is I actually go out on the internet. I find out when sunset is and I figure out how overcast it is and I determine when the lights are gonna go on. As I said before, X10 also already supports a remote, so there's a bunch of remote applications. This is one of the cooler applications that I've done. We actually seven up from the house so my aunt lives with us and everything so when aunt gets up, she kinda sets the coffee maker and then when my wife comes out of the shower, she presses that button and the coffee maker comes on and then by the time she comes downstairs, the coffee's ready. I'm not a big coffee guy, drink tea, but she loves it, my aunt loves it. In fact, it's so funny cause my aunt'll get up and she'll kinda set the coffee, put the water in and everything and then she'll plug it in, but there's no power to it, right? And then she'll kinda go back to her room or whatever and when my wife presses this button, it'll turn the coffee maker on but the problem was that my aunt didn't always know when the coffee maker went on. So there's a separate X10 chime in my aunt's room. So when she presses this button, it turns the coffee maker on and it makes that special chime in my aunt's room so she knows that the coffee has now been turned on and she goes and pours it and does whatever she does and that's her thing that she does and she loves it, okay? If you thought that wasn't crazy enough and this is kinda how it works, you hit the item, it turns on for, I think 10, 15 minutes, I think only, it says 30 minutes, it turns for 15 minutes and then it turns off again because you don't wanna burn the coffee, right? So it turns it off for 15 minutes and then it turns off and that's usually enough time. If that's not crazy enough, my wife has an icon on her phone, on her mobile phone or some smartphone, okay? It's connected to XvConnect which is an Android application that does SSH with secure keys and so forth and effectively what it does when you press that button, it makes an SSH connection to the house, it runs the coffee command from the command line and then it logs out, okay?  It's the kind of thing that wasn't that hard to do, it makes things easier for them, sure, why not, right? This is something you do year 10, okay? This is not something you do year one but as your family gets more used to it, you start to get into these type of applications and again, the house starts to do cool things for you, okay, that you may not kind of get used to doing. So it is pretty cool. So this is how it works, the control of a smartphone uses XvConnectBot, runs SSH, doesn't need a password because you're using an SSH key and it runs the coffee command from the shell prompt and then logs out, not that hard to do. Once again, all these pieces together, all of a sudden they work really well. Pool pump, I'm gonna cover this a little bit. The problem with the pool pump is that it uses a lot of electricity and you don't have to run it as much when it's cold out, when it's hot out, you gotta run it a lot. So hey, we'll just go out on the internet, we'll find the temperature outside and based on that temperature, we'll run the pool pump for two hours or four hours or six hours, okay, automatically. Just go out on the internet, what's the temperature and then just control it that way. And you can control it yourself, you have shell scripts you can control and so forth if you're doing something special. But again, I'm not sticking my hand in that border thing with the dial and those pliers. And also the, I don't have to move those little like knobs and get it to the right time and just all control. Found the activity screen, we actually have a screen that shows like everything that's going on, so it's like a slideshow and then the calendar and then all sorts of activity. Again, when something comes out of the weather, if somebody calls on the phone, if there's an event, you have all sorts of stuff sort of all together in one place, okay. We have an eat button, which is basically a button that you hit eat and then a broadcast, I talked about this, tells everyone it's time to eat. I'm gonna give you the last one, which was actually my most hardest one. This is one I stewed about for a long time. My wife was like, when somebody comes home, can you tell me when they open the garage so I know they're coming home? Instead of them just walking in the door, like why can't I know when a car is arriving in the garage? This is really hard. And it took me a couple of years to get to the point and I think just walking through this will give you some understanding of how bizarrely complex this is. Again, I'm running out of time so just give me two minutes to wrap this up.  You could do it with lights, motion, distance. You can do it with activation of the garage door, garage position. But you also have to take into account that somebody can be taking the trash out. That's not a car arriving. And you also have to realize somebody could open the garage door to leave or they could do it to rise. It's the same action, okay. So one thing I thought about was like maybe using like a detector. So here's what I ended up doing. I took the garage door, I put a switch up there near the top of the garage door and X10 controller. And this allowed me to identify when the garage door was open. So this is the garage. Yes, it really is that clean, believe it or not. My son, my Peter put up his shelves which are fantastic. So basically you have a garage. You have three garage bays here, okay. And right up here, you can see there's like a little switch.  Actually this is probably a better shot. Okay, so this is the garage and here's the switch. And right here is a little, you can see a little bar. It's hard to see but there's a little bar there. And then that connects to a switch which connects to an X10 control which connects to the power. So when the garage door goes up, it actually, and it's hard to see, it actually trips the switch and the X10 identifies the switch has been closed and it actually sends a signal to the X10 monitor that identifies somebody came in. The problem is you don't know if they open the door to come or go. How do you fix that? Well, you could use some kind of sensor, maybe Arduino distance sensor. And I was like, this is gonna be really complicated, really expensive. So I ended up going like super simple. See that up there? That's a door sensor. See that up there? That's a magnet. When they open the door, there's a wire that goes from here along the wall into an X10 sensor. So I know when the door has been opened. If they open the door and then they open the garage door, they're leaving. They're leaving, they're gone. The garage door opens and nobody has opened this door for five minutes prior to that. They're coming, okay? So it's the kind of case where it took me two years to think of it. I didn't want to get into a whole complicated thing, but it actually works. To put icing on the cake, somebody said, it would be neat if we knew who was in the car when they came home. Well, I'm like, everyone's got a cell phone and every cell phone has a MAC address. And the way I set up my network, every cell phone, every MAC address for every cell phone member in the house registered to a particular host name where I put a little DNS comment in there of whose cell phone it is. So when somebody leaves, when somebody leaves five minutes later, I'm like, you know, I had 10 MAC addresses 10, five minutes ago. Have any MAC addresses disappeared in the past five minutes or 10 minutes, I think? And when I identify that somebody's left and a MAC address has disappeared, they were in the car. And when they come back, I'm like, I don't use that in that car because I know who was in that car when they left. And I basically say, Van has returned with Christine. And that's like the newest thing that I had kind of worked in. So conclusion, this is, yeah, I know I'm four minutes over, but anyway, yeah, this is sort of the scary of why you don't want to animation. But as you can see, if it's done right and it's done in a balanced way, I think it can be really powerful. And I think you as engineers, certainly have the potential to do that if you want to, and if you feel that your family would appreciate it. I know mine certainly has. I've enjoyed implementing it and I know they enjoy using it. So thank you very much. I'll be up here for questions. That's it. Oh, perfect. Okay. Okay, so. Yeah, in a few seconds, let me check something here. Yeah, let's go. Welcome. Thanks. Well, first of all, I'm very excited. This is my first scale and I got my first batch, so I expect to be coming every year. And it's been an exciting event. So people are very nice. A lot of DevOps, CISAD means, and developers, great. So, and this topic is very important. I think that for everybody, not just DevOps, not just CISAD means, but also for developers. Because we as developers generate many problems to DevOps. Right? You know that. So it's like sometimes I'm creating my own application. This application is logging some information and the next day I change the format and at some point the DevOps has to create a new script and we have 10 applications, 20, all of them different with different formats. It's a mess. So on this talk, we're going to talk about our experience developing a solution to fix this problem, to make things easier for developers, for DevOps, and of course for the real business. Fruity born long time ago, like four years ago, it's not a long time, but before to continue, I would like to take some time. If you want to send some comments or this presentation sucks, it's good or not, please use both hashtags, okay? We'll have some teachers, people who's trying to give us some feedback during the session. So we'll be really appreciate. Well, let's get started. My name is Eduardo Silva, I work for Treasuredata. We are the company who's behind Fruity, but Fruity is fully open source as many other projects that we have. We do cloud analytics. So when you do cloud analytics, you say, okay, we are going to manage data for a bunch of people, but how do we collect the data? So that's where Fruity borns, right? So and Fruity as our projects, like a Fruity in both are fully open source under Apache license. So this slide will be available later so you can grab my Twitter blog or projects that I'm involved to. We're going to start talking about logging. Logging, it's pretty important, it has many advantages. For example, if you can get what's application status, you can perform debugging. When you get some issues, the first things that you get is please review the logs if you have it. So and find some anomalies. We come through a chute and of course a logging can be done locally or remotely. When things start scaling, at the end it's mostly remotely. But for a business point of view, logging is important because it helps you to take better decisions, right? Sometimes maybe your application is not working well for you as a developer, for you as a DevOps, but for a business perspective, that could be affecting customers. So things get serious. Logging is not something optional. Nowadays, logging is a must. And there's a few assumptions that first of all, we need to start working on it. When we start doing logging, we usually do it on the file system. And everybody says, ah, let's do some logging here. In this part, I have enough space, right? Your hard disk will never be full. Ah, you got it, it happened to you. And sometimes you say, no, I'm writing this to the disk, but the message that I'm writing is too short as it will never block. What means block that an application is running? At this point, I'm writing to the disk and I supposedly be wait for that function to return. But sometimes it's saying what second, two seconds, three seconds, okay? That means blocking. At our point you say, the local messages that I'm writing and human readable, what that means that everybody can understand it, false. It's very hard to understand local messages until you are in a iteration, in a daily basis, you understand what is going on. And without conduct, next day maybe you have an intern, maybe next week a new developer and everybody's writing new log messages. And you want to filter that information at some point. That's become a problem. And of course you say that your logging system will scale. It doesn't matter. We as a company build this great app. If we have 100 users, we'll be fine. Nobody cares. Sometimes your application is scaled, get viral and you get 1,000, 2,000, 3,000 users. And guess what? What's happening with your login mechanism? It's trying to get stuck. It starts to get blocked. Well, it should work. But sometimes it doesn't. So let's talk about the concerns that we have. So if the login formation increase means that our data is increasing. If we have different message formats, it's more complex to parse the data. Even if you write, you invoke some system code and a practice system label, say please write this message to this file and that returns, that does not means that your data was fully stored in the hard disk or the SSD or whatever, right? Because the kernel need to flush that data for the kernel buffers, right? Now, if your application is running multiple threads at the same time, right? Everyone of those threads is trying to write to the file system. Of course, you need to log, put some mutual exclusion between them so you can write the messages. But you are looking and looking is not good. It solves the problem, right? But it does not help you to scale. Now, if you have multiple applications, what happens when you have multiple logs? If you have multiple applications, it means multiple logs. If we put that application on multiple hosts, it's a mess. So there's a point where you cannot manage the information. It's getting very complex. So login matters. It's really beneficial, but it needs to be done right. There are many solutions. I'm not saying that we have the better solution of all, right? We're talking here about our experience by the community plus the company customers. So when you think about logging, you can think that you have many input sources, right? If you have some web applications, you can think that maybe your Apache web server is generating logs. But in a common environment nowadays, you don't just have just one application. You maybe have Apache. You have an NGINX as a front end doing caching. Behind that, you have a PHP using fast CGI. So we already have three things. And each one is generating logs, logs, logs. Plus, you have your custom application which is made in any language or scripting who's generating their own logs. So from an administration point of view, how do you manage to look at this information and get some result of this? Some statistics. What's happening if you get an error, for example, the customer or the user who's with this mobile app is using the app and get some error, where that error happens? On the front end with NGINX, on the back end with Apache, on the PHP side or my custom application. I think that you are getting the point. So not too far from here. This is what we do, right? We have a bunch of scripts running on Python, Ruby, Perl and all of them are running through Chrome tab or something similar. Because we want to get that data and push the data back to any kind of service. Like for example, we may want to make some archiving. We use Amazon S3. Maybe we want to do some big data stuff. We want to use Hadoop. So we want to push the data to the Hadoop file system. Or some relational database like MySQL or NotSQL like MongoDB, Redis or whatever. So this is a current scenario that most of companies get a problem for companies. I was in the flu indeed booth and we come up with many people. Some of them say, oh yeah, I hear about this topic. And others say, yeah, this is what we do. And this is a problem because I waste hours. And that will not happen on these times. So other thing, okay, you can get the data but you have to parse the data. That is quite expensive. One of the most expensive things when working with them, working services and data is part strings. Because everything is a string here, right? Just a few people is writing logs in binary format. Trust me. So how do we solve this with high performance? So you got the problems. The first problem is, okay, we had different inputs or sources of data, each one on a different format. Okay, I get them. Now how to bypass the data? That's when flu indeed born. Flu indeed born on 2011 to solve all of these kinds of problems. With performance, with flexibility. Not trying to compete with everybody but just to integrate with everybody. Well, as you say, as you know, flu indeed is an open source data collector. That means that it collect data. It allows you to unify this data at some point and push that data back to any database cloud service. This is what we have before. So with flu indeed, we aim to have this. Something more clean and something that you can use in a real way. This is a real way we have more than 1,000 users. Well, what about the future of flu indeed? Well, it's high performance. I could be lying. Try yourself. Build inter-reliability. We don't want that you lose data. There are some solutions in the market which collect the data, but if they cannot switch the data back to some database, if the node is down or you have some outage on your network, the data is lost. That could not happen. That's why flu indeed have some work around for that. We manage structure blocks. That means, for example, if you have a web server log, for example, where you get the IP address, you get the date, you get the meter, the URL, and all those files from HTTP request, what it does is take that information and decompose a kind of message in JSON format for you. That means structure the information at some way. It's like making a lot of key value store from each row of logs that you have. It has a pluggable architecture. The community had built more than 300 plugins, plugins to read from syslog, from syslog engine, for Apache, Nginx, Mexico, everything. And what's internal architecture in a global overview? Starting from here, from the left to right, of course, we have the inputs. The inputs plugins get the data in. Then we have the parsers because we need to parse the information. At some point, may you want to filter that information? You don't want all of them. Maybe you just want to get the whole request, HTTP request that you're getting from the states. Maybe you want to discard everything that's coming from China or India, for statistics reasons. Then, that the data is filtered, it's buffered. That means that, indeed, when it's collecting information, it starts buffering, like for seconds, you specify the time. So once it's buffered the information, opens time out, it's flushed the data back. Backward to an output type. So the basic thing here is when it works with inputs and outputs. And of course, it supports some formators. Formators means a way to refilter the information that's going out, the format. So the internal simplifier, how it works. When input plugin gets data in, okay, and it's filtered, then it goes to the buffer. But the data, it's pretty good because it just adds a timestamp here. Times that means when I receive this message. This is no longer just a log file. It's an event. An event has properties, right? It has time, it has attack. What means attack? Because I want to identify internally of FluentD where this data is coming from. And then you have your records in a structured way. And then it's filtered out, of course. So as an input, maybe you can start FluentD, not just come parse log files. It also can behave as a web server. So I can say, I'm gonna start a FluentD with an input as HTTP server. And I'm making all my apps pushing logs to my FluentD. And then FluentD insert the records back to a database. Or just, you can listen for syslog or syslog ng. We have plugins for everything. And the output, of course, you can send the data back to a file to Amazon S3. If you want to make some archiving, it's quite extensive, but it works. Maybe your local instance of MongoDB, because at the end, what you want at the end is to query that information. And what we do, when we do buffering, we can do it on the file system or in memory. Nobody use memory, trust me. Everybody who's in production use the file system because if something happens, nobody wants to lose data. So you could try it, but it's up to you. And when the data's going out to the output plugins, if you get, for example, a mech of data, one mech, what it does is split small chunks because at the end everything is a record with a time, with attack, right? Small chunks that are slushed back in parallel if you want. So it tries to optimize, increase the throughput through the network. Well, this is a more complex part. Here we have the record, right? We have the time, the record information. It goes to the internal rotor. Do you remember that we have attack here? Attack, I can say, for example, if I have one fluendee running, I can say fluendee. Fluendee, please listen for log files from Apache on this path and also listen for a nginx logs on this other path, right? But what's the difference? I say please, for everything that is coming from Apache, add the attack Apache. For everything that's coming from nginx, add the attack nginx. And then when buffering and splitting them back before it goes to the output, the rotor will say, okay, everything that is coming from Apache, maybe I want to push it to elastic search. And everything that is coming from nginx, I wanted to send it to Amazon S3. So you can split and copy the data inside when fluendee instance as you want. And of course, we try to solve this. This is a common pattern. You have many inputs, many outputs. It always goes for buffer, filter, and routing. It also supports simple forwarding. Simple forwarding means receive the data and insert the data back. That's the basic stuff of fluendee, right? And now we're going to explain a similar configuration file here. Source means from where the data come from. The type means the plugin. The tail plugin looks for files and start reading for new data that is coming into that file. It's like tailing on bash, but through the script. I'm saying a path from where this data come from. I'm specifying the format of the data. So you don't need to tell fluendee how to part the data. Just use the right format that already exists for you. And at attack, back in that Apache. We're not going to focus on this one. But then we have a match. A match means that for every data that match the tag, bacon, with everything, right? We are going to insert this on a MongoDB database, which is called fluend, and the collection name is test. The collection is a specific name from MongoDB database. And also here, well, what we have here means forward. That means that when we can listen for events coming from another fluendee, because maybe your architecture is quite big. And maybe what you can do is make them talk each fluendee to each other. So here we have a more difficult example, where we have many sources and many sources running their own fluendee. Here is working forwarding, because we are forwarding the records to a central or aggregator fluendee, who later can insert the data to any place like Trisha data, Sirocco, Amazon, Google, everything. And also, we have the common case of a Lambda architecture. Are you familiar with Lambda architecture? Okay, a few people, okay. Sometimes when you manage data, and you want to query that data, you have two ways, right? One is using real-time queries, right? Because you have a bunch of records and you perform real-time queries under a set of data that is maybe three weeks from four weeks ago. But when you have data from a year, two years, the queries over that is a bit complex. So you need a different mechanism to parse that data. So this is Lambda architecture, where for some things, you distribute one data to some real-time engine and other to some big data engine. Here is an example, we have Elasticsearch, which is pretty good for real-time, and we have Hadoop, which is really good for MapReduce and that kind of queries. And how that can be implemented in the configuration. If we look at the source, it's pretty much the same that the previous one. Now, look at the matches that we have here. The type of output that we have is copy. What it means copy, that we are taking for each record, we are making a copy for two kind of stores. The first one is for Elasticsearch. We are using the lockstach format, and then for each copy, also send it to all the Hadoop file system, which is located on this hold, this port, and this path. So when I'm trying to explain here, is that does not matter what kind of data you get, you can store it as you want, and that is quite good. Now, who is fluent in production? Well, line, green, a slide share. This was not intentional. I don't know what happens here. Trust me. Really, you know what, because this is out of the record, because we have been talking with the Microsoft people, and they are using fluent in a project. So I just added the image before to came here, and I don't know what happens. Really, I apologize for Microsoft people here was not intentional. I will fix it before top of the slides. This is life, you know, things happen. Well, so Google Kubernetes is using FluentD too. So FluentD is not just to collect data, push data. It aims to implement and unify it log in the year. So hold my whole architecture in my company, where I have different kinds of data, how I can unify everything. That's where FluentD comes in. And of course, we, at the company, we use it a lot. We do one thing for analytics that pay the bills, and we make everything open source. And FluentD is one of them. And with FluentD, we collect around 1 million events per second. That is like 100 of tweets around the world per second. Of course, we have multiple FluentD instances. I cannot share more details because I'm restricted on that. But it's quite good, and it works. And what happened, well, we're talking about servers. We're talking about mobile applications. But they are different scenarios. What about the internet of things, right? Devices connected to each other, or most will not as embedded. IoT is like the new market name. The IoT stuff is growing a lot. It's growing to the order of the billions of devices. We care about different things, about connectivity. But at the end, all of these kind of devices are doing the same, sharing information, sharing data. And they're generating logs. The difference is that logs are not being stored in most of cases in the file system because it's very restrictive, right? But they are being dispatched to somewhere. So they need logging. When we talk about internet of things nowadays, there are two consumptions, very big, where different companies are behind them. Because companies say, okay, I have my IoT devices, but I need to partner with other companies. So everybody's trying to merge to have a complete ecosystem for them. So on one side, we have the old scene alliances and the other open interconnect, each one with their own implementation. So each company, it's creating their own framework. On one side, they're creating IoTivity and in the other, all joined. That is how the devices communicate each other. But it needs logging. So how to collect the data properly from these devices? Of course, FluentD is not suitable for that because FluentD, you need at least 40 megs in memory to run it properly. And you cannot waste that kind of memory in a really small device. That doesn't matter if it's too cheap, you can't. That's where the new project born, which is called Fluent Bit. We just make a big release this week. I don't know if you read linux.com website. We have a full post about Fluent Bit. And Fluent Bit was a solution based on the experience of Fluent D reading from scratch, but in C language. Fluent D is made in C plus Ruby. This is made fully in C. And of course, it's fully open source. There's the Twitter, if you want to follow us, great. So Fluent Bit allows you to collect data in many ways, collect, but also if you're a developer for embedded applications, it's allowed you to dispatch the data to different outputs, same as Fluent D. And it's made for any kind of services to collect data from sensors, signals, radios, we support XB, operating system information. When you have your system running, your embedded system, you want to measure sometimes how much CPU is being consumed because think about it, embedded power consumption is an issue. So you want to see how much I'm consuming, maybe I'm consuming the half of CPU time. And that is really bad. You want to get that decrease. And of course, it can run on telematics or automotive stuff. So when we think about Fluent Bit at the beginning, we say, okay, how it should be developed. So we say in C, it must support plugins, right? And must have an integration with Fluent D. Because maybe you have a full architecture, maybe a lot of IoT devices generating data, but you want to merge that information at some central point. That's why it supports Fluent D. So this is a generic solution, right? Well, Fluent Bit works. On this case, Fluent Bit resides on the embedded device. This is just an example. Where the data source can be an IoT framework as the one described previously, an XB device, an XB is a small radio device that's in the market, or some other Linux peers. Once it gets the data in, it flushes the data to Fluent D. And of course, after that, you can flush to everywhere. This was really good. People was very happy, but we get a lot of feedback from embedded people. Hey, I don't want to use Fluent D. People from embedded is very strict about some things. So we started supported direct output to different kind of services. What you see here, there are actual output plugins supported by Fluent Bit. The thing is, of course, we don't do battering on the file system yet, because of its reason, but some customers are asking for that at the moment. They say, it doesn't matter. I am using a really small board, but each one has a one gigabyte for the file system. So please use 100 megabytes for that. Recently, we had support for the Elastic Search. So when you collect metrics for embedded or any kind of system, you would like to have some visualization of that data. Elastic Search is very good for real time queries, but also provides a very good tool, which is called Kibana. Do you know Kibana, right? Cool. And Kibana allows you to make some graphics about your data to make it some sense. So we have the support for that, so you can make some graphics of the CPU usage of your system. Now, we are going to jump to another context, container. Who's from here is using containers? Who's using Docker? Ah, yeah, well, Docker is the same. Docker works on top of the Linux containers. It's pretty much the same. Not the same, but it provides the right interface. So when you run containers, means that you're deploying applications, not just once, maybe multiple times. So we have here about use cases where people have a hundred of containers or more. Cloud services, they have maybe a thousand. So how do you collect the logs for those applications? So we make a deal because Docker on its version one that six implemented the login layer. They knew it, that login was very, very important. So we just come up with them and said, okay, we can build the driver for Fluent D natively for Docker. So one of our colleagues just did the driver and after a lot of iteration and work, this resolution was merged. What that means that now starting from Docker one that eight, the Fluent D support, it's there. So we wrote a go driver for Docker. If you get the new version of Docker, it will be there. So if you deploy your application, you can use the login driver Fluent D. You specify the tag, and then it will flush the data to a Fluent D service automatically. And who's using this right now heavily? Open Shift. People from Red Hat, Open Shift, the new versions, they are deploying everything with Docker and Fluent D because Docker solves the problem of containers, how to manage them. And they are using Fluent D to manage the problem of logging, how to manage them. And the Docker output is pretty good because when the container is working, it's split out some information. Like the tag, you know that each container has its own ID. So you get that information, you get the time and the source from this data comes in. Oh, container ID, container name and the log message. And each container often generates many of them. So if you're writing an application that's gonna be deployed on Docker, just stream out some messages if you want for facility to the standard output and your set. You don't need longer to take care about logging. And at the end, you get this many Docker instances flushing the data to Fluent D. And for people who loves Node.js, I added this because people often ask about this. They said, okay, I'm deploying my really cool Node.js application. Oh, I managed not logging. So we implemented to a package which is called Fluent Logger. It's already available on the NPM package server. So you can get it and start using it. You just create the instance, configure where you Fluent D, the Fluent D tag, where the Fluent D is, the timeout, so every one time it's gonna push this data out and then just one line to send the message. So if you have an environment where people are creating Node.js applications, just try to use this because you are going to unify the whole login of the whole applications. Well, that was my show presentation. I hope you enjoyed. I would like to know if you have some question about it. To modify. Okay, are you asking about when I get the data, how to modify it? If I can modify it as, oh, okay. Well, there are two ways. When you get the data in, okay, you can specify one format. If you change the format, you can change the format inside Fluent D. You can say I modified these fields. And also for each record of event that you get on Fluent D, you can modify it. On the filter process, you can say, please append this key value to the message, which is a pretty common use case. So, and if you have a custom kind of message where the plugin does not exist, you can write your own regular expression for that. So you always can deal with data, but not binary data. But we can listen for TCP, UDP, or LOX, LOX files. Oh, so sorry, please, continue. Yeah, exactly. People will use, they create their regular expression, and you can assign some key names to each value that match the pattern. And people use it to, for example, they have sometimes an NGINX as a front end, and behind that you have many address for different applications. That they are back in microservices, whatever. So they want to filter what is coming to which site and not to aggregate everything in one place, instead on several ones. Yeah, right now in the version 0.14 that is coming out in a few weeks. We are testing that a lot because change it like the protocol at the internals affect thousands of users. So yeah, the solution is there, we are testing it. It's not failing yet, but wait for the 0.14. What? Clustering. Yeah, you can make a copy, a match, and a copy to different FluentD outputs because they're going to talk with the forward protocol, which is called. So that's the big difference. Our solution in the market allows you to get the data and insert the data somewhere. But if it fails, it cannot do anything.  And you can send to many FluentD and this FluentD is gonna get the record and I'll switch it back to 20 place. Yeah, sequence number on table. Sorry, I don't have your question there. Here. Okay. They are not reassembled. So, okay, let me explain. When you get each record, it becomes like a unit, an event of data. Right? No, no, no. What we split is a number of rows, like to say a number of events. If I have 100, maybe I split between 10, 10, 10, 10, 10. And why is that? Because we need to start chunks. When something happens, we send a chunk. If the chunk failed it, we need to retry. Or maybe we have some parallelization with different threads and pushing the chunks between them, through them. Thanks. Okay. You have a question, somebody from them. Anybody else? Okay. Your question is about the performance on throughput. Yeah. The performance, this is a generic answer. The performance always depends of your data. It's not the same, you receive data in a fixed format from TCP that getting data over the file system. Because each step is adding a small overhead. We do our best to collect data as fast as possible and make a reliable system to deliver that data as fast as possible. But if something happens, it will take some delays. It's near real time, but it's not real time. So I cannot say that this will work really, really, really, really fast because it depends of your data and what's the configuration. Maybe you're inserting the records into your Hadoop file system. But if there's a problem in your network, you need to retry some TCP packets or something. It's complex to answer, but our experience says that we get good results. That's why it's been used in production. And it's been replaced, so it is replacing all our solutions in the market. And that's where, and the very good thing is that it's fully open source zone. If you get some performance problem, we can fix it. Each one get the address once it starts. So it can balance and the setup allows you to define which kind of balancing method are you going to use. So it's pretty flexible on that. So we're not trying to just send to this one and get married with that, but it will balance as you setup the configuration. You can set the run drawing and there are other methods. Yeah, the password happens on the input, okay? For example, I didn't mention this, but internally we use some data format which is called message impact. Have you heard about it? Well, everybody knows JSON, right? Okay, JSON is a string format. When you want to parse a string format, it's quite expensive. And JSON is expensive because you don't know where each key start and where each key ends. The same for the values and what kind of values. Internally, and for forwarding, we use message impact that was created by the Fluentd creator which is a binary version of Fluentd, sorry, of JSON. So this binary version said, okay, this field start on this way, it's a map, it has 10 keys, that's it. And you can jump into it. So when the messages go between Fluentd and Fluentd, they use the forwarded protocol and the forward protocol use message impact. So it's not going to use the same parsing that at the beginning. It's optimized for performance. And Fluentbit does the same. Internal Fluentbit is just message impact. Yeah, it's different. Bison from MongoDB is pretty much similar to message impact. The difference I have seen personally, this is my own claim, that Bison is just used by MongoDB and a few others. The difference is message impact, it was widely adopted. You can see we have bindings for Python, Ruby, Perl, C, C++, not binding for C, but it's made in C. So it's widely used. But it's pretty much the same idea, that it's a binary JSON. Escribe. Escribe is a commercial solution, right? Escribe provides you everything to solve this. Most of people switch to Fluentd because of cost and because of flexibility. Because there's some point where you're using Fluentd, as we're doing right now at Microsoft, that they said, okay, we need some extension. So improve that is a bit hard. In Fluentd, as it opens an open source project with full documentation and a lot of contribution, it's easy to extend. If you know a bit of Ruby, you can write easily a plugin. So I think that it depends of the person. I don't want to say that this is times better because it depends of the use case, but I'm talking about what we have seen with customers and with people around the community. Okay, he asked him what needs to be on the input to use Fluentd. The input can be anything that generates some kind of log messages. For example, Apache web server, generate logs, ngnext, generate logs. MySQL, generate logs. Maybe you can hook up your own system, syslog, and let syslog to put data to Fluentd, or syslog ng. On that server, you need at least 40 megs. And all the dependencies are, comes from Ruby gems written in C plus Ruby. So if you go to the Fluentd website, you're going to get the source code of Fluentd, and also you get some instructions. For example, if you're using Ubuntu, long-term support, or Debian, you don't want to build Fluentd and get the whole dependencies. In set, you get what is called TD agents. TD agent is like a package version of Fluentd for enterprise servers. It's pretty much the same, but TD agent is a long-term support version. That is for free too. If you are serious, people use TD agent. Okay, thank you so much. And... Thank you. If you have some questions, there's a booth, booth number five tonight. I have some stickers here if you want to get out some business cards. And... For the people who made some questions, please come here. I have some t-shirts. Test one, two, three. Great is cool. Okay. Hey, everybody. There's only a few of you. Cool, thank you very much for coming at this late hour. Yeah, I just have something to show, which I hopefully you will enjoy and think it's cool. I think it's actually cool. Sorry. Let's do it like this. I just can't put them on my ears. My glasses are in their way. Okay, so what is Crate? Crate actually is a database which is developed in Austria. And as you can hear from an accent, I'm not... I'm one of the few people in the team who are not Austrian. I'm German. The other guys are really Austrians, which basically means if they talk Austrian dialect, I can't understand anything. So we are supposed to speak the same language, but it's sometimes not the case. Crate is not very known in the US. I guess that's as well for the late hour, and this is not a lot of people like here. But I think it's actually a really, really cool project, open source project, database project. It actually won a TechCrunch, disrupt this start-up battlefield contest. And yeah, I'll just let me give you a little bit of an overview. And yeah, so the title of the slide is Big Data from the big Austrian mountains. So yeah, let's go into it. So what is Crate? Crate is a database which was built to scale massively. It is a database which is, so it's not in memory database, which are right now very popular. It's a database which has for sure cache and memory, but where the main idea behind it is to have a database where you have persistent storage. It has very powerful search capabilities and it actually can do very powerful data analysis in a large cluster. And currently, for example, the cluster sizes we have deployed is, so we have a couple. So this is still a very young product. So there are a couple of projects we now offer. A lot of projects we don't really now offer because they simply use the open source version of our product. And we don't really know what kind of scale they have. From the production systems we now, one of the bigger ones is about 120 nodes running on hardware and with about 3 billion inserts per day. So that's kind of something like the scale where you can get to. So it's a really scalable database. The reason, by the way, so I always kind of mix it up or have difficulties with it. So it's kind of, when we're talking so about Crate, there's kind of a little strange thing. So I basically say Crate is built on NoSQL technology on the NoSQL architecture, but the database understands SQL. So this is kind of a little special case here. So we have kind of a lot of features you will recognize from NoSQL databases, but it understands SQL and that's why it's a new SQL database. One of the targets of Crate is to be extremely simple to operate. So it's a database which is specifically built for easy operation and especially to attract DevOps people and developers because you just can simply install the database by yourself and to a certain kind of degree, most probably many dozens of nodes, you don't really need a dedicated database person for it. So you just can more or less run it by yourself. Crate scales almost in linear fashion when you add new nodes and it has the search capabilities. So basically the queries, they scale with it. You can run Crate on commodity hardware if you want to do that. It will for sure scale better on basically server hardware, SSDs, a lot of CPUs and so on, but it's actually doing very well in scaling and you can very well scale out horizontally and in commodity hardware there is no centralized storage or whatever needed. Basically it's a shared nothing architecture, storage is all locally. And Crate is extremely elastic so basically adding new nodes or taking nodes away. This is something which doesn't really require a system administrator to add a new node or to take it away. So you just can simply create, we'll figure it out by itself. Or you just start a new node and Crate will even figure it out by itself. So it will automatically, the nodes in the cluster will detect the new node and they will start to stream data to the new node and just give it something to work. It's very resilient. So as long as we have so we have in Crate we have all the good stuff like sharding, partitioning, replication. So as long as you choose a replication factor which is basically big enough the cluster will be extremely resilient and even if, for example, let's say you have a replication factor of two and you take two nodes down. So then, for example because accidentally you just actually rebooted the node when basically the data will be for a while, some of the data will be not available and then when the nodes come up again the data will be available. So even that is just not killing the cluster. It has read after write consistency. So what that means is it's a typical node SQL architecture. We don't really have transactions but you just have atomic consistency and when you actually write in Crate a row to the database and you just after that immediately read you can be 100% sure that you just get the change row back. So let's read after write consistency. So when you look in the database solution space then, I don't know if you notice but there are a couple of reports every year coming out about basically new databases. They're right now most probably a couple of hundred different databases existing right now. There's a wide mix of databases and different kind of storage technologies, different kind of data models which are behind it. There's a huge variety of databases you can actually choose from and a lot of them are as well open source like Crate. So to give you an idea where Crate is actually located so even if Crate is based on elastic search, so it's basically I will show you later some slides so we didn't reinvent the wheel we just started basically and reused some other open source projects. So even if elastic search is here at the document store we see Crate specifically for Crate and cloud usage. So I'll later show you why that's the case. So our problem right now is we are really drowning in data so there has been never so much data generated as in our time today and at the same time what we are doing what do we really do with the data? So we have the problem that there's a lot of data is generated but we are just not really able to analyze it in a proper fashion. So there's an estimation, this is by the way something I have to give credit to Cassandra or to DataStacks that's numbers I had actually from one of their slides. So the estimation is that there is about a set of data in our world today which most probably is about 135 gigabytes of data for every person on this planet. So what are basically the challenges in this kind of environment? So if you look at the basically traditional data and kind of compare the properties we have here then so with the traditional databases I have most probably my database sizes are ranging from gigabytes to terabytes while when we talk about big data we are talking about petabytes or exabytes then with the traditional databases I'm talking about the centralized database while very clearly with petabytes of data this is not possible anymore so you need actually a database which is a distributed database that's what CREATE is then another challenge is that in the big data space while we have in the traditional data space we have a lot of data which is structured we have basically our table structures which not very often change in the big data space we very often have unstructured or semi-structured data change from one day to the other from one record basically to the other that's as well something CREATE can manage and in the old traditional data space we have stable data models which can be very deep in the big data space that's a problem so we are actually trying to have flat schemas because otherwise this is not really manageable very often the relations we have in the traditional sense they actually resolved as embedded relations in the data itself so that's how this is actually working in the traditional data space I have a lot of complex relationships in the big data space I have fewer interrelationships so a lot of interrelationships are basically embedded then in the big data space I have real-time a lot of real-time data I have analytics I want to do and I have search in general one of kind of stuff like the patterns we have noticed in the last couple of years is and this is something we have seen over and over again is that a lot of people in internet scale projects they actually use technologies so they basically use a mix of technologies for data storage so very often they use a relational database in combination with the document database in combination with some search functionality they need and then they have additionally they have a blob storage or a storage basically where they store their web assets for example so people build different stacks here with React, Solar Rados or MongoDB, Elasticsearch BritFS, CouchDB Elasticsearch and HDFS with Sadoop so this is something which makes projects really complicated because you just have different data stores and that's basically what Crate is actually trying to do so the target for Crate is that you can combine this kind of technologies and that you can have a database with NoSQL capabilities with actually a flexible data model so kind of like in a document of oriented database to store your data with blob storage and with search in one single open source product so that's what we actually trying to do so this is very quick, I want to show you how easy it is I hope you can actually read this so I wanted to show you how easy it is to set up Crate so this is one basically another target we have as well so this should be the easiest database we ever actually set up these instructions are for starting an instance on Amazon EC2 so this is here from the EC2 command line this is starting basically I think this is an Ubuntu image on Amazon you log in to your server then this command here is loading a shell script from the Crate server which is analyzing what kind of Linux distribution it's just going to install you can do that as well manually if you don't want that someone is executing that you're executing a script which you haven't reviewed so there are instructions on the website and local repositories to your Linux server and then it's just going to install Crate at this point I'm sorry for the color here at this point Crate is already installed and you can access it from the dashboard and when you want to start to create a couple of tables then you need to install a basically a command line tool which has the nice name Crate Crash so this is actually the Crate shell called Crash this requires Python when you have this actually requires Python PIP so the Python package manager then you install Crash then you can call Crash and that's actually as I notice this command is missing so it's simply the command of Crash and this is the Crash command line so you have installed and at this point you're connected to your database and you can write you can carry the table you can insert data in your table and you're good to go so this is basically all what's required to install Crate and get it working when you want to install a second server then you basically repeat exactly the same steps here okay if you are in a subnet or if you are not in the cloud so if you're locally in a subnet which can which accepts multicast traffic then the second node will automatically find the first node and the database you have or the table you have created at this point will sync up with the second server automatically so they find each other and they will build together a cluster on Amazon there's one step more required if you want to set a cluster with multiple nodes so Amazon doesn't allow multicast messages so we have a plugin which either allows you to use singlecast or even more elegant you can at the time when the EC2 instance has been created you can give the EC2 instance a tag and then you can define in the great configuration that all the servers which are running in your account which have the same tag are part of your cluster so what's happening is that every couple of seconds your server is scanning using the AWS interface to scan for other new servers with this tag and then it automatically connects them together so the impression about so the syntax for the SQL syntax for create there are a couple of additions so one of the things I was mentioning before is that create is allowing for basically semi-structured data in semi-structured data you can actually define so either you can define this here explicitly and for example doing something here like an object which then basically has nested attributes in it or you can simply start for example create a very basic database schema and then you just simply start to insert data into it and at the time where you start to insert data, create will automatically will first notice the columns which it doesn't now it will modify its internal structure of the table and it will learn the table will basically learn the data you're just inserting so it's adapting the metadata of the table structure this is default behavior if you don't want that because for example you don't want if you have false data or faulty data then this data is creating new columns and this behavior is off but the default behavior is if you insert data then this will create is going to adapt the table metadata also you see here a couple of keywords which are going into sharding and partitioning so I'm defining here how many shards I want to use and I'm defining on which column I just want to do the partitioning also at this point I'm defining how many replicas I want to have in my create cluster so replicas here in this case mean number of replicas two means that in three that in the complete cluster I have in total I have three copies of my data so for example you can simply insert data so you can do a mass insert one so for example one data source you can use is json so if you have this kind of json file then you just can simply do a copy from and then it will insert it into the database so create actually tries to figure that out by itself and it kind of stuff like rebalance on the size of the partition but as well for example you can define here you can define ranges so you can explicitly define what is part of a partition and how many how the petitions I actually use so the the petitions are so all the petitions you define here are as well in all the shards so the petition basically defines the range of the values so it basically defines that the bucket where the data is so it defines the petition that's correct let me actually think about it and I'll come back to you at the at the end it actually maps so because it's actually sitting on top of it this is actually mapping to a Leslie search it's an elastic search term correct so create is actually written in Java so as elastic search is written in Java you will need to have so Java is going to be installed as part of the the process one little information here so that's something which I noticed yesterday with installing the latest version of create so the latest version actually requires now Java 8 so when you install that you need to make sure that your Linux distribution has basically Java 8 otherwise you will need first to install it from somewhere else manually because then the whole installation process will be not able to get the right Java version there are different drivers available for programming with create for different programming languages I'll talk a little bit later about that these these drivers talk with create either with through arrest interface or there's a binary interface and I think binary interface is mainly for Java clients there is a UI with create to see basically the class status it will tell you how many it will give you an overview about all the tables in the system it will give you an overview of where the system is regarding load and it will also give you an overview if there's any kind of problem in the system for example if there's a note down if they are just under replicated then in this case it will show and it will show as well which kind of action it's taking there is a blob storage which you can use so there's an interface for uploading and managing blobs these blobs are as well so managed with for example the replication settings in the database they are as well used for the blobs so they're just going to be stored according to the to the replication settings you have by the way they are not stored in elastic search so they actually stored in the file system of the nodes and it will just distribute so it will actually look for the size and the available space and then it will distribute it across faster there is a plugin infrastructure which is used that you can extend the SQL queries so it's kind of, it's not really like stored procedures but you can for example define, you have something like user define functions and that's something you can implement with plugins and there are many different clients that are available for all the main main languages like Java, Python, Ruby PHP, Scala Node.js, Erlang and as well a couple of other ones also Crate runs really really well in the cloud and as well in containerized environments the main reason why it runs so well there is that kind of like the whole discovery automatically also when nodes are just going up or down so Crate basically takes care of the replication and that the data is distributed into the entity nodes so this are a couple of features of Crate it actually supports I would say about 95 percent of what SQL I think it's SQL92 is actually supporting additionally it allows that you do have arrays and nested objects so that's something I showed you with the Crate table statement so this is by the way directly mapped into nested objects into Elasticsearch then internally it has an information a meta schema which you can query and where you can find out more information about your basically like other SQL based databases where you can find out meta data about your database schemas cluster and node states this is as well something you can find in this meta information so you can build your own little client which is querying the cluster and finding out in what kind of state the cluster is in case this is important for your application and one really really big feature is that in many cases you can reuse your OR mappers you're already using because Crate is accepting most of the standard SQL you can use OR mappers like for example hibernate and SQL alchemy and active record and PHP PDO and so on you can directly use that and if you for example have a project which is running already let's say with Postgres or with MySQL then if you don't really use stored procedures then in most cases it's relatively easy to move it over to Crate so this is a brief overview of the different components on a Crate node so basically all the nodes in a Crate cluster are exactly identical so there is a shared nothing architecture and all the nodes are basically are equal there is though there's one it's not really an exception it's basically there's one node in the cluster which normally has a special use case that's the so called master node and this master node is basically the node which is having all the clusters metadata in it and this master node is something which is not a single point of failure so if this master node goes away the rest of the cluster does an election and elect a new master node so this information they made the information from the master node as well something which is replicated in the cluster and which then can be replaced by a new node which is taking over this responsibility the clients in general have the choice either to use the rest interface or to use the binary interface then we're just using a single parser I will show it on the next slide which is actually so that's Presto which is coming from Facebook then this piece with the analyzer and the distributed analyzer and planner that this is again a piece from Craig this is doing the distributed execution and then it's just merging on collecting the data and just giving that back to the client and for the case of the blob storage that's working a little bit differently so that's kind of bypassing most of this so this is then directly going so this is retrieving the blob directly from the nodes so this is an overview of the different layers and as well what kind of technologies are used here so the base storage layer is Lucene and Elasticsearch so everything is basically stored there and also on the side we just have the blob storage which is the blob storage which is outside of Lucene and Elasticsearch then this is the network layer where we are just using NETI so for basically transferring the data here then we have the aggregation level this is something so you can see from the color coding here where the things are basically coming from so this is what we have developed completely by ourselves this is the distributed SQL, the distributed reduce and the data transformation here for the layer with for the querying we are using Facebook Presto and then this is actually using calling the query planner which is then doing the execution also there is on this layer is the module for importing and exporting the data and as clients we just have so the create dashboard then the shell the create shell which is called crash and also we just have a lot of different kind of client libraries and also Java which is kind of a first class citizen here because it actually uses here the binary protocol which actually uses directly the scene yes so the current version from the the latest Elasticsearch version which right now in the last print was published is using Elasticsearch I think 2.0 that's basically brand new then the second one was what was the actually I know that there is some brand new functions regarding geo location or geo data type basically in there but I have not actually played around with it so I'm not really sure to what kind of query that is supported but this is a feature which came actually in with the latest version so I would need to have a look and see what exactly is supported there so this up the technical highlights for create so it's very easy to scale and to manage database it is based on a shared nothing architecture you can use real time SQL it supports environments with high availability requirements and you just have a data store where developers have a very easy time to develop with because it doesn't really give them a lot of restrictions so you can very freely basically insert data into the database and manage it so regarding scalability so they have been in our bigger setups we had actually up to 300,000 records inserted per second into a create cluster and if I remember this was even not on hardware so I think this was actually on an easy to cloud deployment shards can be actually moved manually or automatically so the standard settings which actually with create come is that create will actually try to optimize that by itself so the so in general data can be semi structured and you can use either strong SQL schemas or schema less SQL schemas so that's basically something which is up to your decision if you choose to so if you basically choose to define a database schema a table schema by yourself or if you actually want that create actually auto detect the schema you can have nested documents so this is basically an elastic search property and these nested fields is something you can use exactly in the SQL queries exactly like you would use normal fields so if you actually have nested fields then you basically have the fields name the nested field dot if there's another nested field dot etc etc and as well the same thing if you have an array then you just can select kind of array elements out of it so this is directly supported with the SQL so this is kind of an extension to the normal SQL that you can have those nested elements there then create the create planner is actually that piece in create which just tries to optimize and have an internal strategy for having the optimized queries on the create cluster the collect shuffle and reduce phases are used for data aggregation across the cluster and for sure create uses a cache for internally speeding up the queries and caching so if you have for example, paging to have faster access here and create is developed with the Java NIO interface so basically it uses the internal asynchronous capabilities of the NIO library in Java to execute or basically process the IO single table you can't you can't have so you can have there are no constraints support also when we talk about tables and kind of some single tables there are joints I haven't mentioned that so far actually in one of the latest versions so they are now inner joints and as well cross joints but for example, outer joints are right now not really supportive so you can have some dependencies between tables not regarding constraints but you can for example do the joints which is actually was a pretty big step so I think it took about a year to develop so it was quite a bigger subject yeah, yeah, you can that's basically happening automatically mm-hmm, mm-hmm no, no, I meant you no, it's happening on the servers so the servers basically no, it's happening on the create level exactly so that's the secret sauce of create basically it's relatively new I have not seen kind of supply good goods so there's I've played around with it but I have not really, I don't really have a test environment where you can see comparison between let's say a MySQL database joining locally versus create actually doing that anyway it's kind of a question what is kind of supply, what is kind of supply good, how can you really test it well because to be fair and having kind of supply good numbers you would actually need to choose a scenario where MySQL really would have problems because of the size so I haven't seen that yet I hope they can publish something on the blog soon about the performance there they're actually a couple of sorry I think there are two block entries right now on the create website which is talking about optimizing basically performance of joins if you want you can have a look there but like I said I don't have a direct comparison between relational versus NoSQL joins sorry no but no we don't I was actually sorry I was thinking about something else so when I was actually thinking about real time data so there's currently some work invested into making create a MySQL slave so regarding streaming data but not in the Spark world I don't know but if you give me your email address I can basically ask so actually my colleagues are meeting tomorrow for some skiing in the Austrian mountains and also they just have a week ahead of planning and so on unfortunately because I really hate the cold so fortunately unfortunately I'm not there so I don't know right now what is planned but I will be able to find out at the end of the week yeah but the problem actually you got it exactly so in general kind of Elasticsearch is kind of packaged into create and you can query create through the Elasticsearch interface but you should not really insert or update data because that's gonna confuse the the metadata layer which is sitting on top of it so the schema information as well the internal information create actually has about it so it's read only the Elasticsearch connection create is licensed under Apache under the Apache license if you want to have a look into the source code then you can do that this is published all the source code is published under on the on the GitHub account also if you want to have a look so we tried actually to reuse as many open source projects we actually could to not rebuild the the wheel so if you're interested in what we are kind of using then you can find that here that basically said just a couple of small words regarding database scaling with create so that's one of the features we are pretty proud about so that you just have a shared nothing architecture so that all the nodes are basically the same the app containers by the way they actually so the connectors we have built here the drivers they are intelligent enough that when they are actually talking to a node that the node actually will basically tell them either to talk with them or basically talk to another other node also there is a mechanism built in in the drivers so in case that one of those nodes basically go away then it actually can talk to another node so that's the in this case the intelligence is built in the drivers create runs really well on different container based environments one of the really really good integration or tests we have done is together with CoroS so for example you can run create containers there with with fleet and you just can actually push the the create containers out with Docker and fleet and then manage them there and kind of scale up and down so this is actually working really really well so in general and all these different environments like container ship directly with Docker or with Tutom so we have actually built Docker containers and we basically have tested in these environments and this works really really well like I said because of great capability of kind of the auto detecting the cluster and then when you basically scale up and down to do this all automatically yes if that's the case then we have a problem yeah sure so I think I mentioned that before so if Mazikast doesn't work then there are a couple of other possibilities to configure it yes what is the default right now Unicast yes so like I think I mentioned it for example in Amazon you have different ways of doing it so one ways I think you can do it with DNS but kind of the most elegant way of doing it is do it with tagging the instance and I'm not sure so I don't think that the Docker container has been tested with ECS there's actually a methods scheduler I think which has been written yeah yes that was about maybe three quarter years ago there's a scheduler for that and so it runs in methods sphere as well I'm not sure about Kubernetes so I think there was something there's something built there as well but I'm not sure about that so the easiest way is just simply search on our website so you will find it we had actually two or three weeks ago I think we had a problem with the block and part of the content actually was down because of a little mishap but right now all the content should be up again and as soon as there's kind of a new thing out, a new feature out you will find it for sure on the block there's also quite a lot of documentation out there it's actually pretty well documented the only thing so we have a couple of third party for example drivers they might be not as well documented but whatever is actually maintained by crate has actually quite good documentation and if not there's in the bottom corner there's a little chat box and there's a real engineer sitting there so you can actually ask them these kind of questions and so the only thing you shouldn't do is most probably do it kind of super late in the night because this is kind of Austrian so middle of European time so they're not going to be in the office but if you kind of try to chat with them in the morning they will basically directly respond to you so this is the crate is available as a docker container so you can directly use that you can use it with swarm then it's on Kinematic so that's another place where you can actually get the container so that's this one here I think the projector is actually not really sharp and also there are a lot of different ways of run a lot of different cloud platforms we can create so there's actually on AWS there's an AMI you can directly use also there is a website if you're interested you can come to me and I can give you the address but there's actually a cloud formation generator and the cloud formation generator you can kind of type in I want to have, I don't know an X large crate cluster with 10 instances and I want to use EBS storage and with two availability zones and blah blah and then at the end it will spit out a complete cloud formation template and then you can load the cloud formation template into Amazon and the only thing which you need to have ready at this point is your SSH key in Amazon and everything else it will automatically orchestrate and create so that you have security groups and all the other stuff will be created for you and configured Azure their image is available for GCE their image is available and then also for soft layer their image is available here's a little bit different it's all basically installation procedure there that's one of the things I haven't tried yet so I can't tell you exactly how that's working Crate is mainly as you can see popular in Europe so we are very present there at user group meetings and conferences and so on so in the US we start to have right now as well our download numbers we start to have more and more people looking into Crate so in Europe we have right now the most people kind of looking into Crate if you want to play around with it I'm also recommending there are a couple of nice examples you can try to kind of get a feeling how you can develop with Crate how the drivers basically work so there is on GitHub is a GitHub example so I don't know if you know but there's so the GitHub has actually opened up there so they have a quite big amount of data about the commits of their users to public repositories and that's actually data you can mine and GitHub has right now every year has kind of a data mining competition where Crate are finding kind of interesting analysis of that data and we have actually built such an analysis on Crate there's an example you can download from GitHub which analyzing languages request latencies sentiments and so on so that's the way the example looks you just can basically reuse for your apps and you can see how the analytics and the aggregation basically perform so if you download the complete GitHub archive this is gonna be 100, 200 gigs of data at least so you can see how this is actually performing how Crate is actually performing with a big data also what I want to mention is Crate is so because there's actually elastic search under it works very well with elastic search tools so for example you can use Kibana for in this case this are some web blocks and this web blocks are actually fed into Crate as a SQL table and then you just can for example use Kibana on the elastic search interface to analyze that as a dashboard not to my knowledge so that might be possible that you can do that through the plugin interface but I'm not 100% sure sorry, what is it? the writes are consistent yeah there is a performance penalty for doing that because you need to wait till basically the last of your replicas have actually written before you can actually return I'm not sure how this is implemented I think this is as well a great thing as far as I know I just only know that you can tune it and for example you can disable read after write consistency but it's heavily recommended to use that okay that's all I had as slides questions correct so when you download a block Crate will automatically so you don't need to know about which node is actually located so this will be handled internally by Crate it's not chartered, that's correct it's basically it's a file which is in but it's actually so it's replicated and you will be for example if this is kind of the next question so it will not for example if you download it if you have a replication factor it will only download it from one node so it will not go to the multiple nodes and kind of stream it at the same time so it will be only downloaded from one okay then thank you very much and so I'm also in the exhibition hall forgot right now which number the booth is something like 300 something so whenever you have time or want to ask some more questions these come over so I'll be there tomorrow at the normal hours cool thank you very much


Hi and welcome to nursing school explained in this video on complications of spinal cord injury. If you haven't already done so please watch the other video that actually discusses the mechanism of injury and the signs and symptoms and treatment and nursing care of spinal cord injury in a separate video. So the three complications here that I'll be going over today are spinal shock, neurogenic shock and autonomic hyperreflexia. So spinal shock occurs immediately after the injury. It can last anywhere from 24 hours up to six weeks and it includes a complete but only temporary loss of certain functions that mostly include the musculoskeletal system. So motor and sensation as well as autonomic activity are usually lost below the level of the injury and that is because the brain is unable to transmit any signals to the muscles and organs because we have that injury and there is no communication below the level of the injury from the brain. And so signs and symptoms include flaccid paralysis, there are absent deep tendon reflexes that might be decreased visceral and somatic sensations that can also lead to anhydrosis which means an absence of sweating and then a paralytic illness because the visceral organs are also affected not only the muscles so keep in mind it's internal as well as external. And the treatment for that includes as with anybody with a spinal cord injury spinal stabilization supporting ABCs and then PT and OT and rehab and then other measures to support the bladder and bowel that I discussed in the other spinal cord injury video. And then neurogenic shock is a little bit different which can occur 30 minutes to six weeks after the injury and it is a distributive type of shock remember there are different types of shocks and the urgent shock falls into the distributive category here and that usually occurs when there's an injury either to the brain the cervical or thoracic spine so high level spinal cord injuries and it means that there is a loss of the autonomic nervous system control to the blood vessels because now again there is no communication from the brain all the way down to the blood vessels and so the patient loses their sympathetic nervous system and the sympathetic tone to their blood vessels which results in massive vasodilation keep in mind the sympathetic nervous system usually helps with vasoconstriction but now we don't have that because it's impaired so it results in massive vasodilation and which will result in in significant hypotension that can lead to tissue hyperperfusion and then impaired cellular metabolism and cell death it's basically low perfusion no oxygen to the tissues means the cells are going to die so signs and symptoms include significant hypotension because of this massive vasodilation but also the hard rate goes down and the temperature might be elevated or it might be low depending on what is going on with the patient in terms of how they can adjust to the external environment if at all now treatment for this includes again spinal stabilization as in anybody with spinal cord injury the administration of vasopressors to help constrict these blood vessels and bring the blood pressure up to ensure the tissue perfusion and those usually include norepinephrine which is also called liver fat or phenolephrine which is also called neosinephrine and then the patient because of the low heart rate here might also need atropine which basically counteracts the influence of the parasympathetic nervous system or the patient might need a pacemaker now this can be so significant like I just said that the patient might need a permanent pacemaker because this can last up to six weeks and we can't just continue to give them the patient now autonomic hyperreflexia is also a life-threatening injury as is neurogenic shock and what happens here is now the sympathetic nervous system below the level in the level of the injury responds to a stimulus of the sensory receptors and this can be as simple as a wrinkled sheet that the patient is laying on and this then leads to vasoconstriction because now we're hyper stimulating the sympathetic nervous system and the reflexes the primitive reflexes it controls and because of this vasoconstriction now will constrict the blood vessels the blood pressure goes up and it can be as high as 300 millimeters of mercury and you can imagine how that can be life-threatening to all the different organs and then there's no opposition from the parasympathetic nervous system because we have that spinal cord injury there is no communication but the barrel receptors that sit in the carotid arteries and then the aortic arch they stimulate the parasympathetic nervous system which is kind of like the backup system in case the the brain of the spinal cord can't communicate and that leads to a hypo brain cardiac but there is no basic dilation because of that spinal cord injury so usually the barrel receptors would stimulate the parasympathetic nervous system heart rate goes down perfor blood vessels dilate but now because the sympathetic nervous system is so overly excited um it it continues to raise a constrict so now we have a problem with low heart rate and high blood pressure so quite the opposite that we have over here with the super low blood pressure and the most common causes for high autonomic hyporeflexia are the standard bladder and bowel and this is something that usually comes up on an exam I'll tell you that right away um and but it can also be due to skin or pain receptor stimuli and like I said before this could be as simple as a wrinkled sheet that's putting a pressure or maybe the patient just rolls over and they inadvertently lay on the on the collides any kind of stimulus that can send their sympathetic nervous system into overdrive because the function is impaired now signs and symptoms here is because of this extremely high blood pressure the patient will also have a throbbing headache and maybe blurry vision as often goes along with high blood pressure there might be diaphoresis above the level of the spinal cord injury because that's still functioning normally and then we have a low heart rate sometimes in the 30s and 40s there might be pilo erection which basically means goosebumps above um and flushing above the level of the injury and the treatment for this life-threatening autonomic hyporeflexia is to check the blood right away if the patient complains of a headache so if you're treating a patient with with a spinal cord injury no matter if this is um right after the injury while they're still in the hospital while they're in rehab or maybe even if they've been discharged and they've been home for a year or two because this can certainly occur at any time and then we want to elevate the head of the bed to help with the vasodilation here and then we certainly have to determine the cause so if the most common causes are bladder and bowel dissension then we have to relieve that by maybe catheterizing the patient or helping them to get rid of any fecal impaction and then removing any skin stimuli such as you know like I talked about the wrinkle cheese or the colic that they might be laying on but also in order to control this really really elevated blood pressure we might need vasodilators such as nitroglycerin, nitroproside and sometimes hydrolysin is also used so in summary the autonomic hyporeflexia as well as the neurogenic shock of both life-threatening neurogenic shock is more a distributive shock where the sympathetic nervous system vasodilates resulting in low blood pressure and low heart rate. The autonomic hyporeflexia is high blood pressure because of that stimulus and that overdrive of the sympathetic nervous system so high blood pressure and then bradycardia and the spinal shock is more involving of the motor and motor and sensory system such as this flaccid paralysis, deep tendon reflexes and paralytic alias so we have to think of visceral as well as somatic innervation here that can be affected. So thank you for watching this video on complications of spinal cord injuries. Please give me a thumbs up if you enjoyed it and I'll see you again soon here on Nursing School Explained. Thanks for watching.