First of all, thank you for coming. I think you all know me, but for those of you who don't know, I'm a very old self. I'm Italian and I'm currently the chairman of the Bavarian Community. This is a speech I used to do really often because I care a lot obviously, but it is also my precise role, meaning that we talk about community and membership, and so this is something I care of. So this is the agenda. I improved my presentation. I told you I had made this presentation many times. So I had the chance to improve it more and more. One of the new features is this kind of agenda which reports the number of slides for each argument. Let's move on. First talk about our community. In our community, there are people from every corner of the globe, and everyone can realize that just looking at us, other people, for example. And who speak and write in so many languages. And this morning we had many speeches, many talks about particular languages. My own system has called it that way. And again, we all know that one of the key features of the open source software, against the closed and proprietary one, is the ability to customize and so to have it translated into many and many just as we wish. With the most diverse competence and skills. I want to remark this because many times people think that software means developer, means engineer or server. But we'll see later on that there are many, many things that everyone can do within the project not being engineers or developers. And we don't... Yes. I have to stick with the last point. We don't have just developers, but many, many kind of. Our community is being inclusive and open to everyone. Becoming LibreOffice users, obviously, is the very first step because otherwise you don't know either what we are talking about. And I must mention and remember that LibreOffice is a free office suite downloadable from the website, which is www.libreoffice.org. And then if you like the software and most of all... By the way, if you like the software and you can see what the spirit which is behind the software itself. So what moves the community may be it's time to get in. How? Everyone usually has some interest, some passion, and maybe you can just put that in our project. Maybe, for example, I used to be a professional speaker and also an EJ. And that's why one of my common plans is to give the official voice in a time project. So when you see official videos in time, voicing this in its mind, but not only for LibreOffice to be honest, I also contributed with a public money, public code, the Italian video as my voice. Because I like it. So it's no hassle, it's no boring for me to do that. It's enough to have just a little spare time, to see that they're willing to have fun with no obligation. We'll see later on. Meaning that anyway, you can donate your time as much as you wish. Free software allows you to do tons of things at near zero cost. For those who doesn't know what is LibreOffice, it's comparable to Microsoft Office, obviously. So it does all the same things. But for example, not everyone knows that there is a module, a component called draw, which is meant to prepare, obviously, every kind of drawings, vectorial, not only raster, and it can be useful for a total of things. I used to, for example, for technical diagrams, because it has layers, so you may overlap many layers, and you can, how to say, switch them out and off. You can decide to show or to print those layers and so on. It has the connectors, really useful for diagrams, flow charts, and whatever. But you can also use it to make a poster, a postcard, or whatever. It's a graphical service, including LibreOffice. And you have to give back something instead of just taking it, because the first reaction to people when you say, hey, do you know about LibreOffice? LibreOffice is a very, very interesting algorithm, right? Please do it, really. But at the same time, you might say, wow, this tool is really useful. It really solves many of my tasks and my problems, so I would like to give back something. My time, for example. Sorry. So that's the spirit of our members. That's what everyone in our community soon or later realize. You don't know what is LibreOffice, but what about the Document Liberation Project? The Document Foundation, which has the main network in Berlin, is the house for those two projects, LibreOffice, which is the more known and the bigger one, I mean, I'd say, and the Document Liberation Project, which is something... Well, it's a piece of code that, I have to say, makes it possible to open and save different kind, different file files. So for example, if you are able, in my LibreOffice, to open and save a Microsoft Word or Microsoft Excel and so on, kind of files is thanks to the Document Liberation Project. So they used to do these kind of filters, we call it that way, which are software that are able to translate the coding of files. And this is how you can be part of our community. The icons here means that the marketing, for example, is something you can do both for LibreOffice and for the Document Liberation Project. The same for the documentation. The translating is just useful for LibreOffice. But what are these areas? This demonstrates that no technical skills are always needed because marketing is what I'm doing right now, for example, so speaking, or using Twitter or Facebook or whatever, and just let the product known, just let LibreOffice be known on the widest audience possible. That's marketing, obviously, and not only that. But the documentation is to write a manual. So it's just about humanity. Humanistic, are you saying? It's just to know English. Well, sorry, where translating is, the ability to obviously to translate from English to native languages, be it German or Italian or French or whatever. The funny thing is that also, if someone is willing to do that, it could translate the product in its own language. So in Italy we have, and also that we are LibreOffice in Venetian. Do you know Venice? We have a lot of slangs in Italy, but a lot of slangs. And Venetian is recognized by UNESCO as a language, not a slang. So they care about translating LibreOffice in Venetian. And this is possible because it's open source. So we may just jump in and do what you wish. Quality assurance is about reporting bugs. So we see that the software doesn't work well. So we just have a final report and say, hey, I've done this, this, and this, and it didn't work. To be honest, before doing this, it's much better to try aging the bugs, which means to test if someone else's bug has the same behavior on your computer. Because maybe just a photo of your computer, of his computer, and maybe on yours, it works perfectly. So it's really needed to try aged bugs. And then you get used to it. You may also start reporting, reporting bugs. There there is the infrastructure, which is, yes, a little bit more technical. And we have Klof here, who is one in charge of this kind of, both with Geo and others. Infrastructures are the website, for example, meaning the server behind the website, just to mention one. But there are many, many infrastructure. Because we have the public website, the private one, and the next cloud, and the gallery, and so many web tools based on servers. And they have to be maintained. That is a sign. This is something really interesting, because, again, you don't need to be a technician or have IoT skills. Maybe you are studying at school, design or something like that, or you're just passionate for it. And you're able to draw new icons or a new interface over there. And you can just apply and make your suggestion. Finally, there is development, obviously. Finally, bliss but not lust. And because it's a core, obviously. And here, you need to be a technician. I told you that this is not the first time that this is going on. So let's go quickly in the very... This is not because I'm Megamani. I don't know how to say the English, but the picture is mine just because of privacy. Because otherwise, I should have asked someone, can I publish your picture? So this way I'm free to use it as much as I can. And to be honest, this was exactly two years ago here in Namibia when I came for KBD Academy. And I was talking about the graphics. And this was an interview completely for God. Back in Burma. I did it, yeah. And let's say two or three years later, you published and I completely have forgotten to do it. Really, it was a surprise for me. And a bit funny also. Then conference segments, training, because you may give some kind of free training, if you wish, once you are expert about the graphics. Promotion staffs and photo presentations will be your interviews to show maybe one day if you want to make a hopefully in TV, in a national channel. Gastarion. And social network, again. Twitter, Facebook and whatever. So what are you waiting for? Christmas? Are you shy? So you can help write memos. Online help. We keep pages. This is a documentation project. And support others. This means that an image of your language is time. But because there are many lists, so new users may arise and kind of question. Say, hey, I just installed the graphics and I am able to do this. And maybe you are not that used, but that thing you are already now. So maybe you are an expert, but for the sake of chance, you just have done it. And so you are able to answer. I just see social networks ask, which is a kind of forum website, where you can also vote the answer, as well as where it is or less. And Facebook and so on and so forth. That's perfect for shy people because they won't have to show any way of thinking. No more excuses left to join. Sorry for this bakery. This is my computer, which is going, leaving us. Let's go to translation. What can you translate? Manuals. The manuals which we are talking about before of the program. There are hundreds of pages for each component. So a lot of work. Online help, again, in your language. Website because it's more appealing to have a website in your own language. The wiki pages, the LibreOffice interface. So when you do file, edit, and all the way, for example, all the dialogues, okay, in Italian it's okay and good, and file orifica, which is edit. And that has to be done by someone. So if you wish to have your LibreOffice in your language, you're welcome. But there's something interesting that you have to specify here because it looks like to translate you have to be spoke in English as well. This is almost obvious, but not completely. This means that in manuals, for example, there are pictures. The pictures of the program. So let's say that the manual says open the file menu and then hit on open, for example. Then you have a picture with the open menu. Well, translating the manual means not only translating the text, but also the pictures. And translated pictures is not tricky because you just have to watch the image and do exactly the same thing and you're copying without knowing even at all English, because you just have to open the very same menu on the same interface or dialogue, whatever.  That's it. So you're translating without knowing English. By the way, also the marketing stuff because the marketing group who is Mike here and Italo, they produce a lot of paper, well, not exactly physical paper, by the way, and a lot of documents, they say, but it's fundamental to translate that as well because if you want to do marketing, the best way to do marketing is to change the language at least in your country. Again, I have a relationship with this about the screenshot. Okay. Then the recent quality assurance. I told you, this is a bug. I reported. Well, it's still open, I guess. I'm not sure. Anyway, I told you you can do bug triaging. This is the first step because before being able to submit a bug, you should first do triaging, some triaging all at once to understand what's the way to do that. By the way, it's not difficult. It's just that we call it the monkey list, meaning that you have to write down one, open the graph, two, do this and this, and then expect a behavior and a current behavior. So you just... But it's a key point to have the monkeys because the other people who has to triage a bug they don't have to waste your time. So it has to be really quick, I open a bug, do this, this, this, this and then the triager has to just write down it works for me or it doesn't work on my computer as well. Then the reporting, the hunting. There are kind of campaigns where in the two days they run this kind of campaign to find the bugs of the new release. So there is a new release who is not yet released but is going to be within two weeks usually and there is this kind of campaign in two days, a sprint to hunt for bugs. There are many little things about the quality of the software. The design is obviously this is an Italian as we may see. It's translating the program, not sorry to say, placing I'm sorry, placing the gadgets which is sorry, so maybe we'll be much nicer if this could be here instead of there kind of this kind of stuff. So the user interface, the icons in the dialogue again but all sorts of ways. Maybe in your office you use some kind of planning, I don't know, particular and there is no template that you create for yourself but share it because some of these may find it useful. The infrastructure as I told you, these are just some of the service run by a foundation which needs to be maintained. This is a little bit technical but no need to be developer but be the agnostic. Some kind of thing about developing and this is finally the coding which starts from macros. I mean, code macros is not real development but it's still needed. Then extensions and there are the easy huts which are the welcome door for new developers for the colors and then the code development and whatever. Even if it is just to be mentioned we also run sometimes workshops in which you can learn how to develop. Why contributing? Satisfaction? Feeling part of a project feeling useful to others which is something that to me at least really gives a lot of satisfaction. Getting touched by the community and the social event just like the conference. Meeting different people and cultures is really amazing from my point of view.  Oh, nice meeting you. Thank you, it was annoying so much. Other beer. I have the beer. I'm just enjoying it just like I'm doing right now. I'm growing in college because you have learned something from others especially in these kind of communities. Using them and membership itself which I explain quickly now as a reference for job application. Fix your goals if you are able to otherwise learn fixed hopefully. Finding your aiming credits and everyone who contributes in any way, just tweeting maybe just using Twitter is then listed in the website as a contributor to make a difference because yes, little things makes a difference in our work. If you like to make people nothing better than dinner. Finally, I just talk about membership one minute too. Yeah, well, okay, couple minutes. All those stuff you can do without being a member of the Dr. Foundation. If you wish to do it you're free to do that. But if you do it regularly, it means not once in a while but neither every day maybe once every week, once every month whatever you wish and you can give as you donate at the time. Then you may be part of the foundation. The difference between the community member the foundation member is the governance. You may vote and be voted if you can for the board of directors or the membership committee which I am chairman at the moment.  Have much fun this is for sure. Get satisfaction be the growth of other projects because this is surely one step more to grow it, to grow that. Get strong and superb because yes we are not paying, we are talking all about volunteering so no money here. But at least as a recognizer of your commitment and your contribution the review for expenses because we are a certification program, as a developer as a trainer as I am and as a migration expert and I am too. Migration means that you are expert in replacing Microsoft Office or whatever the company or the entity is using with the legal office and find your job because many I can be sure many of our members have found their job within the community and I have one in front of me for example many because because you learn many things and you get into an ecosystem which sooner or later will need someone to hire us as a direct DTF employee as a contractor as a self-employed ecosystem like I am or an employee for our company involved in the project to become a member but I have already already told you so I would like to thank you thank you for attending really leave your questions really quickly


Hi, I want to welcome you to Postgres versus MongoDB for real-time machine learning on wind turbine data, where a speaker will discuss why turbine systems chose Postgres over MongoDB to capture and analyze wind turbine data in real-time, and how they run machine learning on Postgres at scale. My name is Lindsay Hooper, and I'm one of the Postgres conference organizers, and I'll be your moderator for this webinar. I'm here with Luis Carril, product owner at Swarm64, and Michael Tichtmeyer, CEO and founder at Turbit Systems. I want to give you a little background on each. So Luis has developed Swarm64's Postgres-based analytics solution for several years, and now helps to solve client problems and incorporate them into the Swarm64 technology. Previously, he worked for several universities and research centers in software engineering, cloud, parallelism, and high performance computing. And Michael, prior to founding Turbit Systems, Michael conducted data science research at the Free University of Berlin, concentrating on the application of machine learning and data science, laser physics, and wind turbine measurements. He founded Turbit Systems based on this research. So that's it for me. I'm going to hand it off to Luis and Michael. Y'all can take it away. So what are we going to do today? First, I will give you an introduction about what Turbit Systems is doing and why we use machine learning to analyze wind turbine data. And then I will also give some thoughts and insights about our database and why we first used Mongo and now are switching to Postgres. And I will give you some insights and wind turbine data into the market, how the market behaves, and why... Yeah, it's a very special market and I think you're going to find some interesting insights there. And then in the end, Luis will talk a little bit more about Swarm64, which is an awesome data-based company that's accelerating Postgres. And we are using that currently. And in the end, we will have some Q&A. So machine learning for wind energy. First, to give you a side, the wind industry is growing. We have more than 350,000 turbines globally installed and also more than 650,000 megawatts in installed capacity. And we all know that because of climate change, we somehow need to ensure that wind energy stays competitive because I think, and many other people believe that this is a good alternative. And in order to ensure that, we need to think about how to reduce costs. And yeah. So to look a little bit more into the cost structure of wind energy, here on the right, you can see different sources of energy, right at the right, offshore wind, biomass, then photovoltaic or solar, then nuclear power, coal, and then onshore wind and water power. And you can see that onshore wind energy, so the height is the cost in US dollar per megawatt hour. And you can see that the cost of wind energy is actually already quite low, which doesn't mean that we need to put it lower. And especially if we want to go into a regime where wind energy is not subsidized any longer and by the state or by other forms of subsidization. Yeah. So let us look a little bit into the lifetime of a turbine. So we have usually in Germany or mainly the manufacturers produce turbines that have to last 20 or sometimes 25 years. And during that lifetime, the failure rate is changing. So in the beginning of the lifetime of a turbine, when the turbine is installed on the side, you have early failures because everything is new. Maybe it's a new turbine model. And maybe you need to install different operating management due to bad control or other animals that are around. And all these typical stuff ends up in a higher failure rate, let's say three years in the beginning. And then after a while, the turbine should run for the next 15 years. And hopefully in a good way and nothing breaks down. And sometimes you have a gearbox failure. You have a really severe failure that you, of course, want to prevent. And then after 20 years, this is actually a pretty interesting time because the turbine is not manufactured to live longer than 20 years, but theoretically can operate the turbine longer than that. And in this wear-out time, of course, you have more failures coming up because just some parts break apart. But if you manage to have the turbine running in that last period, that's actually quite valuable in terms of the financial perspective. And of course, in all three stages we want to reduce the failure rates. So here I have a study from the German Fraunhofer Institute to give you a little overview about what failures can happen in a turbine. And here you see in red on the left side, you see the frequency of a failure in a typical component. So here you see the components. And then in blue on the right side, you see the downtime for this particular failure, the average downtime. So just to give you the names, so this is electrical component, then electrical operation, sensors, hydraulics, the yaw system, yaw motors, then the rotor blades, then the mechanical brakes, the connection between the blades and the nacelle, the gearbox, the generator, and the nacelle and the drivetrain. And what is remarkable is that you have maybe here a very high rate of failures for electrical components. But what costs the most? That's the question. So if you, for instance, have a gearbox failure, it might not happen very often, but the downtime is quite high. So you have like six days of downtime here in this example. And that means that turbine is not producing energy for six days. And if you have a three megawatt turbine or even more, that means a lot of money. So if a gearbox is failing, you need to order maybe a gearbox, you don't know if that gearbox is stored somewhere, it needs to be transported to the turbine, then the crane needs to be installed, everything needs to be managed. That's like a lot of stuff is happening if a gearbox failure is occurring. And that's the reason why we maybe don't want to concentrate on the failures that happen so often, but we want to concentrate on the same failures that produce the most costs. And in this case, this would be, for instance, the gearbox, the generator, and the drivetrain. So maybe also another thing that one needs to understand is that there's a difference in the operation of a turbine between planned maintenance and unplanned maintenance. And so every half a year, technicians need to go onto the turbine and need to do some, yeah, planned maintenance. And apart from that, every additional time, technicians need to go on the turbine, it's additional costs. So you want to reduce these unplanned failures and make them into planned failures. So you want to know if a failure is occurring and maybe the gearbox oil is heating up and or maybe there's not enough gearbox oil in the gearbox that would be super bad. But you want to prevent this and you want to pre-plan the maintenance with good knowledge from data analytics. Another thing that's also interesting right now is that everybody on the market wants to reduce costs. And that also means if you have additional hardware installed like Barbation sensors on the gearbox, that means also additional costs. That means that this additional sensors need to be maintained. And there's a new data connection. And you add a lot of more work if you also add more hardware into the turbine. And that's why we try to use the sensors that already exist on the turbines. So just a short overview about turbine systems. We also have some hardware where we detect vibrations. But our main focus is on the data analytics of the so-called SCADA data. I will explain what that is later. And we do with that data, we do some machine learning. And so in the end you can call that SCADA Condition Monitoring. A little bit more about the wind turbine data. I think if you are coming maybe from another domain, for instance, if you have a water power plant, then you have maybe one plant. It has seven similar turbines inside and the water is flowing down. And everything is the same. High volume project. And in wind energy this is totally different. Every turbine is different. It is standing on the different side. It's standing. It has different components. It has different gearboxes. And so that's really, really special. So if you have 350,000 turbines in the world, then you also have 350,000 different plants. So each and every turbine is gathering these supervisory control and data acquisition data. And theoretically you can log one second values, even sub-second values. You can connect to the brain of the turbine, let's say. And standards like newer turbines have up to 500 plus metrics. There can be, for instance, of course the wind, the power output, but also like the turbulence intensity and temperature of the air, temperature of the gearbox, pressure of the oil of the gearbox and so forth and like many, many values that you can look at. And also status logs where maintenance teams, when they go on the turbine, they log what they did on the turbine. And as I said, this theoretically you can get more data. But the norm is that especially like wind energy is like quite old, let's say. They started with 10 minute average intervals of these data because they didn't know how to handle so much data. As I said already, you have many different data sources, you have many conventions and regulations, especially in Germany with nature protection, which is good, but it makes things more complicated. You have potentially a lot of data if you want to log everything. You also need to think about okay, do you physically, does it make sense to log the temperature every millisecond? I think that doesn't make sense. But for instance, if you have motion data of the rotor, there might be actually some information inside that higher quantization of the data. And then additionally, every turbine is special because of the site. If you have a turbine in the middle of a wind park, it is of course behaving differently than if it's in the front of the wind parks, it has less turbulences, and also the height plays a role because of the air pressure. And this is the reason why if you want to apply machine learning on this kind of data, you really need to think about the physics and really need to think about okay, does it make sense to have one model, one machine learning model for all of the turbines standing everywhere, which is that we believe not the case, but we need to make many machine learning models per turbine and per site. So this is why we have this, this is kind of the structure of our data platform. So we gather weather data, then of course the sensor data of the turbines, everything that we can get. And then logs, additional logs, maybe from the ERP system, like financial data that maybe gives us some more insight. We collect everything in our database. And then apart from that, we generate digital trends and we offer the solutions or the insights in different products. For instance, we can offer a trained machine learning model of a specific turbine type so that people can understand even better if their turbine is behaving, let's say as the fleet of that turbine type behaves. So you can compare your own turbine with other turbines in our database, then of course predictive maintenance, but also you can compare within the wind park performances and many things. And also of course we sometimes deliver custom solutions for our customers. So this is maybe for wind farmer, this is the power curve. This is the basic thing that wind farmer always looks at. You can see here on the x-axis, you can see the wind speed. And on the y-axis, you can see the power in kilowatts and the wind speed is in meters per second. And you can see for instance that here the turbine sometimes stops because maybe grid operators are regulating the turbine down because the grid has too much electricity. Or because of other things, bad control or load reduction. And then you can see that here after 3.5 meters per second the turbine starts running and then it has a maximum because the manufacturers didn't build the turbine to turn around infinitely fast. In blue you see the real values that the turbine has measured. And in red you see the predictions that we have learned or let's say we have trained a model to learn the behavior of this curve. But actually one also needs to understand that this, if you would only plot this graph, if the physics would be in such a way that the power only depends on the wind speed then we maybe wouldn't see so much of a spread of the data. What happens here is the power output is also dependent on the pressure of the air, so the energy of the air. And that depends on the temperature, the pressure of all the density which you can measure with these other parameters. And we use these parameters to train the machine learning model. So actually this graph should be something like a five-dimensional graph. And we are much better than just predicting like a bent curve here. Here you see these predictions in the time frame. On the x-axis you see the time and on the y-axis you see the power. In green you see the predictions. And here over that you see the blue and the real measured data. And in orange we detected parts where something is behaving weird. And you can see here that in February 11 the turbine has been throttled and because of something that the turbine owner didn't maybe want to happen. And then we give an alarm and we can say, hey, a turbine owner, look at your turbine, something is wrong, you should better do something. Another example, a very quick example, you see here again the power and on the y-axis the pressure of the gearbox oil, you see these many states where the turbine can be in. You can see these different behaviors of the turbine. So the pressure of the gearbox oil, this is a normal behavior. And here the pressure somehow has dropped in blue, the real and green the prediction. So something has been wrong here and actually some of the oil cable connections had been broken here and that's why there has been a problem. And of course you want to know that. And you want to know that before the turbine is going into some weird status logs, you want to proactively see, okay, the pressure is starting to drop, I need to act now. So why did we use MongoDB originally? It's exactly because of this different data structure, like we didn't know in the beginning, okay, is this the data structure, maybe we need another column, maybe we need to delete it, maybe, yeah, when you want to develop something on an unknown data source, then I think a non-relational database is a very good choice, especially also if the data changes while you are developing some code, you are very flexible then with MongoDB, but if you go into real time, then you get to some limits or if you want to get into production and get faster and get more and more data, you come to some limits. And as I said in the beginning, we see a high potential in this one-second quantization, meaning that we have a continuous monitoring of a lot of data, maybe even more data points that the turbines are offering right now. And in addition to that, we also want to send data back to the turbines, to the control algorithm of the turbine, to change their behavior in real time. For instance, if you have one turbine at the beginning of a wind park and there's a gust coming in, then of course the first turbine realizes, oh, there's a gust and then it can behave according to that gust, send the data back to Turbot and Turbot is generating a prediction of how the gust would develop into the wind park and tell the other turbines how to turn and how to pitch their blades to have an even higher efficiency. If you want to do that, you need to make the step from 144 rows per day to 86,400 rows per day. And that's about 600 times faster. And for that, you need queries that are definitely sub-second. If you want to go to the second regime, you need to calculate stuff in the sub-second regime, of course. And then you also want to add a lot of turbines and a lot of sensor data and a lot of users. So that kind of gets tricky. But also when you have a large database, how you perform the developing at the moment, especially for debugging, you use a smaller part of the database versus if you really want to get good debugging, you can use a lot of data at the same time. So if you're faster with everything, you can maybe debug machine learning algorithms with a larger amount of data. Because everything happens that you couldn't think of in this data. So you really need to have real data in order to debug machine learning code in this domain. Yeah, and of course, yeah, if everything is faster in production, that's nice. Also, if you have less storage problems with so much data, and as I said, faster development with faster queries, we ended up sometimes with Mongo, we ended up pre-calculating some queries because they were too slow and then saving them again in the database, which now we don't need to do with Postgres and accelerated by Swarm any longer. And that also means, okay, if I want to explore some data very quickly, I can just write the query and I get the data in a nice and fast way. Yeah, and the bug I talked already about. And this is maybe just a quick overview about the packages that we're using. So we have both a Mongo and a Swarm accelerated Postgres database. For the machine learning stack, we use quite simple packages, Python, Django, for the framework, for the web interface, Pandas, Numbia, TensorFlow. And in the visualization, Grafana is quite nice because you can then put real-time data into movement on the screen, which is quite nice and plotly is also good too. So yeah, now I would switch to Lewis. Thank you very much, Michael. Yeah, so Torbid wanted to scale with the number of users, wanted to scale the amount of data, wanted in general a faster database, and to help him solve these database issues, it came to Swarm64. And who is Swarm64? Swarm64, we are a company located in Germany, in Berlin, and we consider ourselves Postgres SQL high-performance innovators. We try to make Postgres SQL faster and in an easy way. We are similar to other databases like Oracle, Microsoft. They have units to research on how to improve the performance, especially for analytics, and we consider ourselves the same for the open-source Postgres SQL. Why Postgres SQL and why extend Postgres SQL? Because Postgres SQL is an amazing open-source database. It's one of the most used, one has an amazing, diverse, and wide ecosystem. And instead of creating a new database from scratch, we try to leverage its advantages. Postgres SQL is mostly focused on high performance for OTP, for transactional workloads, but in Swarm, we are more focused on analytics on bigger datasets, hundreds, if not terabytes of data, long-running queries, and so on. So our main product is Swarm64 DGA. DGA stands for Data Accelerator, and we are in the version 4.1. That works for Postgres SQL 11.7. Swarm uses all the extension mechanisms provided by Postgres from hooks, front-end wrapper, custom scam providers, custom SQL functions, C backend functions, to offer greater parallelism, to offer a custom storage engine to reduce IO consumption, and even optionally to use an FBA as a coprocessor. How Swarm64 DGA makes Postgres faster? As we can see on the left side, Postgres, since Postgres 96 and Postgres 10, especially, Postgres is able to parallelize queries, but Postgres is still quite conservative in how much parallelism it introduces in a query, and at some points, during a query execution, it cannot longer execute the query in parallel, so it has to serialize it. Swarm has a much more aggressive parallelism, has reduced IO costs with helps with the parallelism, and not only that, it has custom execution nodes that helps a query plan to be executed in parallel for much longer. So a longer and a deeper parallel execution pipeline leads at the end to much faster runtime execution, and as we can see, the CPU can be uploaded to the coprocessor. Swarm64 DGA contains several features, and the core one is an optimized storage, and this is hybrid row column format in opposition to the standard row format usually Postgres, and here what I mean is really how the rows, how the data is stored on disk. While Postgres writes complete row one after the other, Swarm splits the attributes, basically the columns, and writes some kind of columns together and other columns together, so when you don't need to retrieve all the whole row for each query. So the data is vertically partitioned based on the columns, but it's also partitioned horizontally in bigger chunks than Postgres pages, and on top of that is doing compression. All of this helps to reduce the, to vastly reduce the IO consumption. And this is transactionally safe, and up to Postgres 11 we implemented that using the foreign data wrapper interface on Postgres. So that means that your queries will not need to be modified, and only some small dvl changes basically to create table statement will need to change. On top of that we have extra our custom range index and some custom clustering. In our development for the foreign data wrapper we also, at some point we needed to create to support backups because it's not supported by Postgres. Sorry, wrong slide. So we also developed a patch and we are contributors to the Postgres community, and we have a new feature coming in Postgres 13. Optionally part of the computation of the done by the Optimus Storage can be uploaded to an FPGA, a field program or data write coprocessor that will take care of the compression of the recompression and filtering of rows that will liberate the CPU to execute other parts of the query or execute or execute other queries in the system concurrently. But how fast is Postgres in comparison with Swarm 648? Here we have a comparison between Swarm, that is Swarm in Postgres and native Postgres using TPCH. TPCH is a very well known analytics benchmark that emulates a warehouse where orders for items are received, the items are shipped, there is tracking of the suppliers and so on. Here we have a test on scale factor 1000 that means approximately one terabyte of data and we execute this benchmark in a DL380. On the left side we can see what is called a power test and a power test executes the 22 queries that compose the TPCH benchmark one after the other giving all the resources of the machine to the whole query. And here we can see that Swarm 64 is 60 times faster than Postgres. On the right side we see the other variant of the benchmark that is in a multi-user environment where four users now again in the same system in the same terabyte of data the four users are issuing the 22 queries of the benchmark concurrently. And we can also observe that still even under such a high load Swarm 648 is 14 times faster than Postgres. So going back to the beginning of our story Turbit needed acceleration and scalability. They needed to improve that exploration how fast can they retrieve the data that they want to see and the data that they want to show to the users. They needed faster reporting response time. They don't want the users to wait many seconds. They needed new analysis that they could use for example for the machine learning for the machine learning algorithms and for that they will need to execute it in huge amounts of data. If we want to go to the sub second we will need to scale and be able to support six hundredths time the amount of data that currently the database was receiving. The Turbit is expected to grow in number of clients with means growing in number of turbines and of course they wanted to accelerate the data retrieval for the machine learning algorithms. So we agreed to, so Turbit came in February this year and we developed a multi-stage plan project to help them tackle these issues. And the first ones have already been moved into production. Here we can see the recording architecture of Turbit where report. So Turbit will report initially was only using MongoDB and to slowly enable Turbit will report to migrate its functionalities to Postgres as they see necessary. We set up a second database using Postgres and Swarm with Postgres where all the data for Mongo and Mongo is still here, the main, the master copy of all the data but all the data for Mongo is continuously streamed into Postgres. So Postgres and Swarm contain a complete copy of the data and we are already offering several services to the Turbit web report using only Swarm data. Services that will not be possible to offer if the data was querying MongoDB, at least not without extra engineering effort as Michael described. We have a better reporting worker funnel and we have two custom analysis. But how is the database, how is Postgres and Swarm set up here? Well, we have to remember that here we are mostly seeing time series data. So the temporal component is the main driver for all the queries, all the analysis. We have a custom mode, the custom module that replicates each change on the Mongo database into Postgres. So we are always up to date and the main design in the Postgres and Swarm is very simple one. It's basically a single wine table where all the MongoDB data is available as a JSON column and any other key attributes from the JSON data or from the MongoDB that is already necessary for the high-performance queries done by Grafana or other analysis has been already flattened into individual columns. This huge table is also horizontal partitioning and individual turbines because mostly each turbine is going to be query alone or we are going to see aggregations between turbines. But here we have already, so the horizontal partitioning has already, it's already a query optimization mechanism, it's going to save a lot of IO. On top of that, each partition is going to be, as we saw, is going to be stored using the hybrid row column storage format on Swarm that is going to execute extra horizontal and vertical partitioning. So really queries that require to retrieve only days or only a month is going to retrieve from this exactly the amount of data that they need. As we see the compression here plays also a huge factor. We are speaking at the moment, we have around one terabyte of data in two bit. This data in Swarm, thanks to the compression, takes 12 times less. Thanks to Swarm64DA, Turbit has now a couple of extra analysis that they can offer to their clients, directly to their clients or to their own internal algorithms. One is gap analysis, as we saw the turbines emit 10 minute aggregates of data, but sometimes they fail to report this data. That could be because of network issues, malfunctions in another place, and so on. So we have an algorithm that explores this time series and finds gaps for each turbine where no data is present. This can be used by Turbit clients to maybe retry and try to recover this missing data. This can be used as an input for the machine learning modeling. It can be used to identify which turbines are less reliable in the sense that they provide less information or we can use the gap analysis to interpolate and generate the missing gaps so we can use other kind of algorithms that require a very constant time step. Another analysis is also downtime analysis, it's a multidimensional statistical analysis to identify the root cause of why a turbine is not generating enough power, because we can get that the turbine is working but is generating a much less power than expected. This could be because of unexpected maintenance. It could be because of a monk function or any other cause, but if we are able to already in the Turbit web report to tag this thing, to tag this root cause that makes more clear to the wind farmer what is going on. The third point that Swamp was offering was faster turbine data exploration, in this case using Grafana. We have something already with Michael, how the exploration of how looks like the power output versus time from a turbine. Now, if we have a lot of users accessing the Turbit web report and querying for data, we will want to have fast response times and keep the user experience very low. Also similarly the same queries that we will need to feed into the machine learning control feedback back to the wind turbines needs to be under a second. So in this graph we can see on the horizontal axis a comparison where an increasing number of clients is accessing the Turbit web report that is they are doing queries that retrieve data and make computations on the database. And in the vertical axis we see how many seconds. The red dots is the time needed by Swamp64DA to return the expected output while the gray dots are the same for MongoDB. And the yellow line, yellow strip line is the average for MongoDB while the black line is the average for Swamp64DA. And we see that Swamp64 scales much better, has a much flutter than MongoDB, is able to be in sub second with the current amount of data so it's able to serve 5.5 more users than MongoDB which MongoDB basically with four users already starts to suffer. So the behavior of Swamp is much faster, not only faster it's much more stable. We see that the variability on MongoDB runtime is much higher than in Swamp. This behavior is not only visible in Turbit. We had a previous project with Toyota about connected cards using post-DAS to locate vehicles, to geographically locate them. And here we can see a similar analysis. Here the number of threads is basically the number of users, the number of queries being done concurrently. In Orange we see the response time for Swamp64DA here, but here in blue we don't have MongoDB, we have post-DAS as well without Swarm. And we can see that also against post-DAS Swarm is again much faster and much more stable and with less variability. This enables you need to scale out less, that is things that you will need a cluster to serve. Now you can do it in a single server, is what we call scaling. So here we have seen what has Swarm64DA done for Turbit, but can Swarm64DA help in your particular case? Well here we have of course several things to consider. Swarm64DA is mostly planned or thought to be used for analytics, where we speak about hundreds of gigabytes, gigabytes of data, big queries, queries that needs to go across to prove a lot of data. We need queries that they need to be able to be parallelizable. In Postgres, depending on if you use too many functions, if you use PLPT SQL, you might find that your queries are not parallelizable. Swarm64DA increases the parallelism, but adding more parallelism to something that cannot be parallelizable is pointless. And Swarm64DA helps also in situations where you have high concurrency. So here being honest, if your workload does not match at least one of these cases, it is very unlikely that Swarm64DA is going to help you. So Swarm64DA again builds on Postgres, is not a fork, is not a new database, is a plugin, is an extension, it builds on top of Postgres. So most of the ecosystem, if not all the ecosystem of Postgres is compatible with Swarm64DA. In Swarm64DA, we have a very simple, very affordable pricing, it's $53 per virtual core per month. Currently, Swarm64DA is delivered with Postgres 11, 11.7, as we have said, it can be also combined with enterprise DB EPAS, can run on-premise, can run on cloud. We are also immediately available in AWS. And you have, in case of to opt in for the coprocessor path, you also have several alternatives to our partners. Finally, if you think that Swarm64DA could help you, or maybe even if you doubt it, there are several ways to try Swarm. You can contact us, you can get a trial license, a lab license, and you can try Swarm64DA in your own data center, or even this year, you can go it to the cloud. We have a seven days free trial where you can have a Swarm64DA server up and running in less than five minutes. You just need to visit the Swarm64DA website and you will find all the information. And with this, thank you very much to everyone for coming in, for attending, and we are glad to answer any of your questions. Thank you, Luis. Thank you, Michael. There was one question that came in that was, I think for Michael, and that is, what do you do when you see gaps in the data? Yeah, so turbine data can be a little bit messy sometimes. So we need to understand that if the turbine, if the grid operator says, for instance, that there is too much electricity in the grid and we need to reduce the amount of power in the grid, then what they do is they can switch off the turbine. That really means they switch it off. There's no electricity on the turbine. It's completely still. There's no electricity there. And in the meantime, also the manufacturers didn't have a solution to have a battery or something to at least measure the wind speed. No, they don't measure that. It's completely switched off. There's no light on the turbine, not working. And of course, during that time, you have no data. So if you want to do time series analysis, you have a problem. You want to fill these data gaps and you need to come up with some solutions, whether to use the data around that event or not. And that's quite an issue. So you need to first identify these data gaps, and then make sure to find a solution what you can do with that. And yeah, that's pretty nice with SWAM. We were developing some algorithms for that to handle this situation. Thank you. Okay, we're at the top of the hour now. So actually one other question is there a white paper on the architecture of SWAM 64DA to understand how it works? The answer is yes. We can send everybody a link to that afterwards. And that explains where SWAM, what use cases form accelerates and what use cases it does not accelerate. As Luis already mentioned, it's really focused on speeding up analytic type of workloads, query intensive. And especially queries that, if you have queries that seem to respond well to parallelism and postgres, they're going to speed up even more with postgres, which makes all the parallelism better. Another question that came in is what additional benefit do people receive when they run form with an FPGA coprocessor? I think as Luis mentioned, if there's an FPGA present on the server, we install an FPGA image that uses the FPGAs processing to add even more parallel processing, about 100 plus SQL reader and writer processes that speed things up further. And generally speaking, it's around a 2X increase in performance on the server. So it's actually quite a good way to speed things up. I think that's it for now. We're at the top of the hour. So we have a few more here, but we'll follow up with those via email. Okay, wonderful. We'll thank all of you for joining us today. Luis, Michael, Andy, thank you for... Does people get when they, do people receive when they run swarm with an FPGA coprocessor? I think as Luis mentioned, it just, if there's an FPGA present on the server, we install an FPGA image that uses the FPGAs processing to add even more parallel processing, about 100 plus SQL reader and writer processes that speed things up further. And generally speaking, it's around a 2X increase in performance on the server. So it's actually quite a good, from a price performance perspective. Are there any further questions, Andy? It's a good way to speed things up. I think that's it for now. We're at the top of the hour. So we have a few more here, but we'll follow up with those via email. Okay, wonderful. We'll thank all of you for joining us today. Luis, Michael, Andy, thank you for being on and presenting. And I hope to see all of you on the next Postgres conference webinar. Bye-bye now. Thank you. Thank you very much, Lindsay.


Happy New Year, shipmates. 2013 was a year of challenges, but it was a year of a lot of successes, and you made those successes possible. And that's why I look forward to 2014. We're going to be just fine. Now, our navigation plan is going to be our guide for 2014, and my resolution is stay on course. Our three tenants will guide us through the year. Warfighting first. We will bring you the capability and the technology to get the job done. Operate forward. We're going to continue that Asia-Pacific rebalance, but we'll maintain the course and speed, if you will, in the Middle East. And be ready. We'll bring you the training that you need while at the same time getting away from those degraders from our readiness, such as sexual assault or substance abuse.  I need you to take care of each other, look out for each other, and be safe. I want you back at work after the holiday period. And be fit physically, mentally, and morally. In this 2014, we're going to be where it matters, when it matters. You will be the difference. I'll see you out there in the fleet. Happy New Year.