All right, so let's get a start. I'm sorry for this initial delay with usual stuff on presentation sometimes. So I'm very happy to see you all here. What I'm going to be talking about today is an open-source solution that we have built. It's called TorDB. And basically, it presents a way of running no SQL loads on top of a relational database. So this is something that can be used if you're, for instance, you're a DBA or you're running a relational database and you would like to offer to your users some no SQL interfaces. So it's more than just having an instructor data. This is something that's already possible on many relational databases, like especially Postgres. It goes far than that. So I'd like to talk about this issue. A little bit about myself. Before I work for a company called 8kData, it's a research and development company on database, the database area. So we try to come up with crazy stuff and this is one of those crazy ideas which we came up with. It happens to be right now our main line of development. So this is our core product right now. I also work as a programmer. So I'm both kind of a DBA programmer and I'm based in Spain, Madrid, where we also founded the Postgres Spanish community which has become a really large Postgres community. We're very proud of that, more than 500 people currently. And this is a little bit about myself. You just wanna connect with me, find me on Twitter or LinkedIn or any place. If you have any question after this talk, I'll be around but just in case you wanna write me, you know, this already coordinates. So let's talk a little bit about this problem that you may face. The world has definitely changed a lot in the last five to 10 years. A lot of things have changed and databases are no special. I mean, in general terms, technology has changed in so many areas. I'm not gonna talk about that. But in the database field, it's been kind of a revolution in the last five to 10 years. There's been like 40 years where the relational technology, relational theory has been developed and advancing and advancing it's has become probably better and better and better, but it was kind of stable. The players were more or less the same, the databases were more or less the same, the market was more or less the same. But now in the last years, everything has changed. Many databases have started to appear. Many new technologies have been introduced. I don't know if you can see, can you see this picture? I don't know if you can see this picture. Me neither. I'm quite close. I can read this picture. I don't know if we can turn off some lights maybe or something. Or maybe those are you who are in the back. Maybe want to move to the front a little bit. I would say that's not bad. You can sleep if so you want. Now, yeah, probably not. Anyway, feel free to move to the front if you want. I think it's gonna be better. You do some extra exercise too. Well, so, well this picture, this picture is just an event and in 2005, there were no lights, there were no pictures taken by by phones and tablets as in 2013 where everybody was with their own phone and tablet. Things have changed lately. So also in the database as I was saying. So let's imagine you were a happy DBA. You were running your stuff. You were running your users. You were even happier if you were using foscrash, right? You were taking some time and you were buffing your users when needed. And so it was a happy life, right? It was easy. The only problem that you have to deal with is basically those programmers who haven't met before Mr. Bobby Tables, right? So it was a happy life. Now what happened? That at some point, no sequel came into place and people start yelling at you, hey, we want no sequel. Hey, we want MongoDB. Hey, my app is web scale. And problem started. Our peaceful life was kind of ruined, right?  So rather than doing all this, why can't we say to your users, okay, do you want Mongo? That's okay. You want no sequel? Sure. You want no web scale? Whatever that is, sure, you can have it. And this is the problem that we are trying to deliver today. And the way to do this is not by, you know, installing MongoDB or Cassandra or any other no sequel database, because that's a huge problem. It means if you need to install a new stack, that means that you need probably new servers, you need new support contracts, you need, you know, probably people trained and certified, you need new backup procedures, new security measures, new firewall rules, new network rules. And of course, once you have done that, you still have a lot of problems. Like how do you synchronize data from one database to the other one? How do you make that consistent? How do you make sure if one user or one application is querying when data source, that the other one is gonna be up to date too? How do you move data between those? How do you ensure consistency? So this is a huge problem. So basically answering this question, satisfying these needs, these requests to have no sequel on your data center, probably you have experienced that, it's a problem. So fear no more. Here comes DoraDB. And DoraDB is basically, you can think of very simply as Postgres plus Mongo on the same place. So no new servers, no new stacks, no new certifications, no new, but even backup procedures. You have all that on the same place and it's open source. So let's put it very simply. What is DoraDB? DoraDB, it's a document database. It's a no sequel database. It's a JSON database if you wanna call it that way. But happens to run on top of a relational database, data store, which is Postgres, and it's open source. And the most important thing is that it is compatible with MongoDB at the wire protocol level. And of course, that means also the API. This basically means that as long as we speak the protocol that MongoDB speaks, there's no need to have, you know, some different drivers, some different tools, the same tools, the same drivers, the same programs that run on MongoDB. It's a little bit of a at risk because of, you know, compatibility, but basically all those work the same in DoraDB, it just happened to have Postgres behind. So, first question. Since MongoDB is non-extractor, this hierarchical data store, and Postgres is relational, how do you map that? How do we store the information into Postgres? Well, this is probably one of the most important aspects of what we've done with DoraDB. This, how do we store the information? And we don't just take the information and store it as a blob on Postgres. We could have done that. But we saw that there was a huge improvement, a lot of benefits of not doing so, and rather transforming in some way the data from an unstructured way into a relational way. So, how do we do that?  So you take a JSON document, and we split that document into pieces, where each piece contains at most one level of nested documents. Because you see, you know that JSON documents can be nested, can be all the documents within those. So we split them into pieces in such a way that there's no nested documents inside a single piece of this JSON field. We call that sub-document, not very original name, but anyway. Now, we analyze the type of a sub-document, because MongoDB or any NoSQL data store, documents in general, they don't have a schema, right? But they have types. Indeed, the MongoDB internal format, the BISN, it's typed. It specifies for each key what is the type of the value associated with that key. So we analyze the set of types that are within that sub-document, and that, if you think about it, pretty much looks like a table. So we have an attribute name, an attribute type, and a value. And indeed, that's a table for us. So we take each of these sub-documents, find the candidate table which has the same attribute names and types, and store it there. Store, just store, yes, the values. What happens if we don't have such a table? Well, we create, we will create the table dynamically. So you don't have to create a schema beforehand. You don't have to do basically anything. You just use an empty database, connect to RGB, start inserting documents with MongoDB API, MongoDB tools, MongoDB programs, and then, 4DB will take care of creating any needed tables, splitting the documents into these pieces, and basically storing the information there. Let's look at an example, because it's gonna be easier. Can you read this on the back? All right. So this is a sample JSON document where it has like some levels of nested documents. So there is one field called name. I don't know if you can see the pointer, anyway, for those who can. There's a field name, and a field data, and a field method. These three form the root level of the document. So this is gonna be one of the pieces in which we will split this document. Then there's another one here in horizontal, called A with keys A and B. A, 4, 2, B, hello world. This is another level with no nested elements. So this will be another part. Then there's another one with a J and deeper, which is inside nested, which is another level. It's gonna be another sub-document for us. And finally, this one, A and B. So basically, the first step that we do is that we take this JSON document and split it into four pieces like this. The root level, which contains name, and then both data and nested, which are gonna be kind of placeholders. They're gonna contain the nested distractors. Then we have A and B, we have J and deeper, which is another placeholder to basically say that there's a nested distractor there, and the other A and B. Now, we find candidate tables for restoring this information, which match this column name and the data type. This is gonna be an integer here, or a number, and the column B and a text field. If the table is not existing, we will create that table automatically for you, no problem. The placeholders, we don't need to store them, because that's basically a pointer. We're gonna keep that information in another place. So this is how it looks like once we store this information into TorDB. We use some extra fields called DID and index, which are not really important, but it's basically a way for us to pull back all the documents together. And then here's the information. MongoDB has an internal underscore ID field, which is a 12 byte integer, 12 byte array, sorry, which is hidden, but it's actually there, so it also gets stored. So this is the root level, where if you remember, we have the name TorDB. As I mentioned, the placeholders need not to be stored. So basically, I just take the name and TorDB on the root table. This is the data table, it's called P underscore three, it's been created automatically by TorDB, you don't need to do anything. Then we have the AMB 42 and Hello World, and notice that this table is gonna be reused, because it has the same column names, the same data types, so both sub-document will be stored on the same table as we have here. And finally, the J42. So this is what we call the data tables. They contain the data that MongoDB document had before. Now notice a couple of things here. This is just one document, if we store like many documents like this one, here in the JSON document, in MongoDB, all this A and B, A and B, and the data types associated are repeated all the time. That's meta information that is attached to these documents. And we are repeating that only once, it's gonna be on the table definition. So this means that the way of doing this, first advantage, we are saving a lot of metadata, we are avoiding a lot of metadata repetition. Now, these are the data tables, but how do we pull back together the document to form the original JSON document that we had? Well, there's an extra table called Structures, and this table, which is very important for us, basically is a tiny JSON document, small JSON document, which basically resembles the structure of the original document. It tells us where is the root level, what are the nested documents, and some other extra information. So basically, this structure says that the root information is gonna be on table number three. To go back to table number three, we see that this is effectively the root level. Then it says that the root level contains another field called data, which is a nested document, which is stored on table number one. So data is gonna be on table number one, we go to table number one, and this is the data. This means null, and there was no index here, which means that this is the data. And then there's a nested field which is stored on table number two. If we go back to table number two, this is what was inside the nested. And finally, nested contain a deeper element stored on table number one with index one. And if we go back, we see on table number one with index one, that this is the other field that was there. So with these data tables and this structure table, we are able to reconstruct the whole document back again to where it belongs in this unstructured JSON format. The good thing is that, again, we were saving before some metadata because it's repeated many times within a normal JSON collection. Now, this structure also happens to be repeated by itself. Documents in a large collection, most of them have more or less the same structure, the same shape, because they are alike. Otherwise, it would be really difficult to consume all the information in a NoSQL database. NoSQL means that you can change the schema, but you're not gonna change it per document all the time because that would be crazy. How would you query that? So a lot of structures are repeated. And hey, in relational theory, we have, well, not in relational theory, but in relational databases, we have foreign keys, right? And that means we establish a relation. So we also establish a relation on this final table, which is root table, which creates an association between documents and structures. So we have many documents which have the same structure. We will not need extra entries here in the table of structure. We'll just point to that structure. That's why we have a field called SID, which is this tractor ID. If you think about it more visually, this is how more or less data is stored on a NoSQL database, basically any NoSQL database stores data like this. It looks a little bit like a mess, doesn't it? Well, it is, because documents are just stored one after the other one. It doesn't matter what information they contain. It doesn't matter how many fields or nets of documents they have. They're just stored one after the other. Sure, there are indexes, but for any reason you cannot use an index. Then you're basically screwed up because you have to do a whole database or a whole collection scan to find your information because information is stored like this. After the process that we do in TorDB, the information is more or less stored like this. Just intuitively, you can imagine that it should be better because we classify documents by their shape, to be more precise, of their sub-document. We have analyzed exactly the data types that they have, the attributes, the columns that they have, and we have put them in separate bins. Now, this is what we call also partitioning by type because we're taking the data and we're classifying it depending on the type of the data and putting that on separate tables. That's partitioning, right? So queries will be more efficient if we are targeting a given set or subsets of document types because we will only be looking at those partitions which refer to that data type, data structure. Now, this is a very typical question. In Postgres, there's JSONB. It was introduced in Postgres 9.4. It was significantly improved on Postgres 9.5. So why don't you use Postgres, JSONB type, data type, this is data type for storing JSON. You know, you use JSONB, right? Well, I guess that's a yes, anyway. So there are many reasons not to use it. I mean, don't get me wrong, it's absolutely cool. I love it, I love JSONB. And it's one of the most important features that came with Postgres, but it's not enough for our purposes because our goals are, first of all, we want to get the data normalized. We wanted to get the data partitioned by type as just shown you, but JSONB will do the same thing as NoSQL does. Store every document one after the other. Doesn't matter what type with structures they have, they will store it in the same way. It, of course, does not provide a NoSQL API. So the problem that a lot of people are facing in the relational world is that they are asked to provide a NoSQL API. So basically, they have to install Mongo. JSONB will let you store an instructor data, but will not run a MongoDB program. TorDB will allow you to run a MongoDB program on top of your Postgres database. Plus, it is, again, it's not compatible with MongoDB and it can also not replicate or chart as MongoDB does or NoSQL does. Thing is that scaling, or isn't all scaling in Postgres and relational databases is too hard because the set of use cases that they represent is very wide and doing a general solution for such a wide use case is very difficult. Now, NoSQL is using a smaller subset of that use case which happens to be easier to scale. It's not freelance, definitely. It has a lot of problems, but for whatever reason, people like it and use it and expect it. So I'll talk about that a little bit later, but TorDB implements MongoDB replication protocol too. So it can replicate from a MongoDB. So you can participate in a MongoDB cluster. That will not be provided by JSONB. And finally, JSONB is tied to Postgres. So far, TorDB runs on Postgres, but it will run on all the relational backends soon. So we want to make it run in different backends too. I already mentioned this. There's a lot of metadata repetition. If you look at the given collection, some documents have the same type, the same shape, and so this creates an overhead. If you use JSONB or NoSQL in general terms, you'll get a lot of this repetition, and this costs you disk space, IO, memory, insert buffers, and so on. So this brings us to what advantages does TorDB have over MongoDB? And the first one is related to this metadata repetition. As long as we are classifying the data and repeating the definition of those attributes and types at the table definition level, which means only one place, rather than every single document, we're saving a lot of disk space. And well, honestly, disk space is not very expensive today, but IO is. And you can trade today disk space and price by IO. So that means that basically if you're using less disk space, you can either pay less or you can pay the same and get a faster IO disk.  So this is really significant improvement. And basically if you look at the numbers here, where we compare MongoDB with TorDB, basically TorDB just requires from 30% to 68% of the disk space that MongoDB requires. And this is not with compression. That's why we're using MongoDB here without compression. This is before compression. This is not compression. This is just saving metadata, avoiding to repeat extra metadata that is repeated all the time on any NoSQL data store. All right, so this is the first advantage. Basically IO. The second advantage. It is surprising that NoSQL is trying to get back to SQL. That's why it's a badly chosen name, right? The problem is that either as much as they are trying to get back to SQL, they're doing a poor job of going back to SQL, which is even more shameful. First of all, because what they call SQL, it's not SQL, it's SQL-ish. Looks like SQL. Like if you look at Couchbase, Nickel and one QL, it looks like a lot like SQL. So developers are good for it, but it's not SQL. So tools are not that happy. Tools basically don't work. SQL tools don't work. BI tools don't work. GUI tools don't work because it's not SQL. So it's good for the developers. Definitely not good for tools. And then even if they were absolutely compatible with SQL, that SQL is just a tiny subset of SQL. It's the basic select where join, you know, offset or something like that, but doesn't go further than that. And we know that SQL is way more than that, right? Especially Postgres SQL is so advanced. So as a friend of mine says, hopefully you're not stuck on Windows 3.11 days. And Windows 3.11 is when SQL 92 standard came up. And so this no SQL database, they are trying to get this SQL-ish language. They are trying to make it compatible with the subset of what SQL was in year 92, which means basically you're not getting a lot of power. But Postgres has one of the most advanced SQL, most compatible SQLs available on relational databases. And guess what? It's free in our case. I mean, of course, Postgres too. But I mean, it comes for free. As long as we're using Postgres as the relational database back end, you get all these powerful SQL. So even though we support MongoDB API and you can insert data with the MongoDB API, MongoDB programs, drivers, whatever, that's good. But then if you want to query in a more special way, if you want to use the power of SQL for querying data, sure, just go to the database. Don't go through our layer per se. Just go to the database and run SQL. Tables look a little bit weird, might be. There are solutions for that I'll present to you later. But anyway, I mean, it's SQL, it's tables. Just go and do it. And it's no SQL-ish and no subset. It's pure SQL. So that's why some time ago, we introduced what we called TorDB views. And these are machine-generated. I mean, TorDB generates this, creates some views which kind of try to pull back together all those tables in which documents were split by TorDB in the process of insertion, and pull them back together so that they look like an entity that is easier to query. This views, so, well, basically, if we create something like this, which by means of varying fields, it will create different tables in TorDB. So these two documents will require four tables in total. And if we look at the views created, it all looks like it's been put together. So it's very convenient for basically doing queries. And in order to create these views, you just need to issue MongoDB command, which is called createViewPath that we invented. So it's not, you won't find that in MongoDB, but you will find that on TorDB. You can create that from your MongoDB program, and then you can use these queries. We also created a new MongoDB program. You can run from the console or from the Java driver or whatever, which is called SQLSelect. And you can just put a SQLSelect in that, and it will go to the database, issue a SQLSelect for you, and return the data as in MongoDB as a JSON document. So you have the full power of SQL at your hands with TorDB. You can even use tools. I mean, tools are compatible because again, it's SQL. It's not SQL-ish. Then we have what we call Toro query by structure. If you remember, we are partitioning the documents by the type that they have. So this partitioning, if you issue a query to TorDB, and that query cannot use an index, then in MongoDB, you'll need to scan the whole database. Rather here, we will look at the structure of the documents that fulfill your query and only scan those tables. So you basically can get a huge performance improvement. It can be as high as the inverse of the likelihood of your document. Anyway, you know, varies from query to query, but it can be really, really, really high. Even negative queries, which are queries that return zero results, they can be resolved sometimes, but just by looking at the structure. So, and the structures are in TorDB, they're a cache to memory. So we could probably resolve negative queries just from memory compared to a whole database that's scanning MongoDB, which is of course almost infinitely faster as much as we want. There's of course the possibility that you can mix and match relational and no SQL data on your same database. Some people requested this and basically there's nothing we can help with. I mean, you can do it. You don't need any kind of support. Just put TorDB and your relational data on the same database. Probably on different schemas, just to make things clear. And don't write to TorDB tables on your own.  But other than that, you know, you can have your own relational tables and the Tor automatically generated data tables and then query both to joins between your relational and no SQL data, no problem. Then there is atomic operations. So MongoDB doesn't have atomic operations. Well, a single document is called to be atomic because it's held by an exclusive lock. So document operations are atomic. But if you do a batch operation, if you try to insert 10 documents at once, it's not atomic. In MongoDB, basically, you have to iterate through all the documents to check whether they inserted correctly. If one fails, you have to probably, and you want that operation to be atomic, you'll need to delete the other documents that really got inserted. But then you'll need to check the results of that deletion because it may have failed too, on some of them only. And then you'll need to kind of reverse that operation too. And it's a chicken, an egg, endless problem, unless everything works well. It's of course not all the time happens. So, I mean, we know the value of atomic operations. In MongoDB, there's no atomic batch operations. In ToroDB, it was hard for us not to support atomic operations. So, you know, it's everything runs on transactions and transactions are atomic units. So, we get them for free. There's another point. MongoDB, there's a huge debate whether MongoDB supports clean reads or not. Well, the reality is it doesn't support clean reads. A clean reads means a read on a consistent view of the database, on the data. I'm not gonna get into the details, unless if you want to, just let me know, raise your hand. But basically, MongoDB runs on read and committed, which means that any new data can pop up in the middle of your queries. And you'll, sorry, is that okay? Yeah. And you'll lose, you'll see new documents popping up in the middle of your query. You can see a document twice on a given result. So, and well, of course, Postgres, it took us just two lines of code, you can see at the bottom, two lines of code to implement clean reads. We basically said, hey, query transactions are gonna run on repeatable read mode in read-only mode. That's it. Clean reads for free. I mean, not for free, thanks to Postgres. It's a great database. This is very fun. You know the MongoDB 3.2, which was released recently, and now support for a connector to connect to BI tools. That's why I said they are trying to come back to SQL because they realized that SQL-based BI tools are better than the non-existing BI tools in MongoDB, or no SQL for that matter. So, they announced this connector. Do you know how it works internally? This proprietary, by the way, connector is not available on the MongoDB open source version. It's only on the enterprise version. But do you know which is the critical piece that makes that connector work? Yeah, it works in Postgres, really. Yes. So, behind the scenes, this connector uses Postgres for in-data wrappers to convert from MongoDB to relational tables. Guess what? That's called TorDB. And performance, well, I can't speak because I have signed a license agreement which basically prohibits me from talking about that performance. But it is dot, dot, dot. Really. I mean, just imagine, there's no pushdown, a lot of pushdown support on Postgres for in-data wrappers. So, well, when for in-data wrappers in Postgres are gonna get better, MongoDB BI connector is gonna be better. Anyway, so they need this connector, it's proprietary, it's a slow, oh, I said that. Well, whatever. And, you know, it requires Postgres. Why don't we use TorDB? There's nothing else to add. I mean, MongoDB, the Mongo BI connector, sorry, the BI connector is not needed, it's Postgres. A lot of tools, most of the tools, BI tools work with Postgres already. So, there's nothing else to do. All right. So, what about performance? Because after all, we're doing a lot of stuff. We're receiving the document, speaking the Mongo protocol, which is not native for us. We're transforming, we're going for tables, we're checking those table axes, we're creating tables, we're, you know, splitting the data out. So, I mean, there's no free lunch, there are no miracles in the world. I think there are a lot of advantages that probably outweigh the sum. Of course, there's advantages that TorDB has.  So, let me be quite upfront. MongoDB, when you look at MongoDB benchmarks, they don't benchmark what you're going to be using in production. MongoDB can have, like, many tunables, and the performance varies greatly from one tunable to the other one. Most benchmarks that you see is running MongoDB in a completely unsafe way, which you will never want to run that way. You'll lose data, for sure. The database may be coming consistent. So, don't look at those numbers. Look at the numbers who run MongoDB on what is called safe mode. Imagine what would happen if you don't use the safe mode. With journaling enabled, I don't see any scenario, unless, you know, for some very particular use cases where you don't want journaling on your database, and with replication enabled. Because, basically, if you don't use MongoDB with replication, I mean, if you use MongoDB on only one node, that's not very interesting, you know, proposition. You better off using Postgres, or, yeah, of course, TorDB. So, when you enable all those things and then compare the performance with TorDB, honestly, with a very patched version of TorDB that was released, not published in GitHub yet, but, you know, released, like, six hours ago, the performance running I.I. Bench, which is a very standard benchmarking software for MongoDB results look like this. Higher is better. The line on top is Toro. Yeah, the reason, by the way, the reason why this line ends up here, that's the question, is because this is just inserting 300,000 documents, so we required less time. So, test is finished here. No. Oh, sorry. So, in other words, we are faster. This is pure insertion. So, if you wonder that, you know, all this stuff is gonna be really expensive, it's not.  It's not us, I mean, we're not smart, it's Postgres. It's really, really fast. So, if you enable, if you disable journaling, or if you disable replication, MongoDB is gonna be faster. But if you use MongoDB in the same way you would use it in production, we are faster. That's simple. This is MongoDB 3.2 latest version, so this is not, you know, an old version, compared with ToroDB running on Postgres 9.5. It's significantly faster than Postgres on 9.4. All right. Let's quickly move over how we're doing with time. All right. Replication. I mentioned that before. I'm just gonna put it simple. We support replication. So, one very interesting use case for this solution is rather than replace MongoDB, which I'd be happy if you do it that way, but it's just replicate from MongoDB. So, let's say some of your users or your yourselves have a MongoDB replica set running already with live data probably used for OLTP. Now, you can set up ToroDB to just lose on the replication protocol, connect to the MongoDB primary nodes or any secondary nodes, which can also serve the replication data, and then have ToroDB start replicating all the data, live, like it's asynchronous, but it's usually fast, and you'll get another copy. And then you can run SQL on that. You can of course run the MongoDB API too, but you know, you can run SQL. So, this is a very nice way of doing very cheap and fast ETL from MongoDB to SQL. Just use Toro. This is, by the way, this is present. It's gonna be released as ToroDB 0.4. We're currently on 0.4 alpha one snapshot, but it's gonna come anyway. If you look at the devil repository, this is on GitHub, anything that's on GitHub, it's open source, AGPL. So, just look at the development branch and all the code's gonna be there, except for this latest benchmark.  So, what about charting? This is where everything becomes really interesting, because we want to implement charting on MongoDB, which means implement the MongoDB charting part of the protocol. This is not done yet. However, and we want to do this in such a way that ToroDB will work exactly as MongoDB on a charting environment. It will talk to the Mongo OS, which are kind of the coordinators on a charting environment, and we'll talk with the other MongoDB nodes to move data around, just to participate in normal charting environments. It's not a very difficult thing to do. We'll come next version. Now, what if we, apart from doing that, let's say again, we're gonna do that. What if we try some of the ways of charting information, charting data, within the relational world, and chart below ToroDB level? Well, we can do that. One option could be like PgChart, the charting extension that Citus data developed in social open source. We're also considering all the databases that are already good at charting at this level, like Green Plum or Redshift or Citus data too. So if we really pair this concept of charting at the database level, we have the concept of replicating from MongoDB replica set, what we're effectively building is a new way for node SQL users to perform data warehousing, data analytics. So basically, this is a technology that will enable data warehousing for, as I say, those poor souls in the node SQL world who are struggling to do this. If you try to do data warehousing in node SQL, you're basically out of game. It's so terribly slow. You won't believe it. It's basically unacceptable. That's why most node SQL users are using tools to ETL from MongoDB or whatever to Postgres or all the databases just for doing the analysis. So with TorGb, you could just replicate the data because it talks to speak the replication protocol, get the data to TorGb and then use a backend which is already charting at the database level. So we did some experiments and we tried Green Plum. It's a great data warehouse. It was presented here yesterday. So we did some benchmarks with that. Green Plum was open sourced late October last year and we kind of already hiked TorGb to work on top of Green Plum. It's not released yet, but we did some benchmarks. The goal was exactly this. To take TorGb, connect to a MongoDB replica set, replicate the information from this replica set to TorGb. TorGb will talk to Green Plum. Green Plum will chart the data across all the segments and then we run SQL. We'll run distributed SQL, we call. SQL that will get pushed down to all the charts, to all the segments in Green Plum terminology and get back to the results. And then we will compare how these results obtained with SQL will compare to those results run on MongoDB. Our rate hold is very slow so you can already get the results. So that'll be as surprising, but anyway. So we took some data set, the full Amazon review data set, which is a data set obtained by basically web scraping, a lot of all the reviews of all the products that are in Amazon. Run on Amazon Cloud, use C for Xlar, Xlar server, which is not a big machine. We want to show that, this is what you may have at home. We set up four charts inside the same host so that network traffic's not gonna slow us down. In MongoDB case, that requires three comping nodes. Comping nodes basically do no jobs, just metadata, so it doesn't add to the load any way they are required to run the cluster. So you have to hop them anyway. And four charts, and four segments in Green Plum. Same size, same memory, same everything. Data set contains 83 million records and 65 gigabyte plain JSON file. So, if we import this into both MongoDB and TorDB running on Green Plum and compare the disk space required to store this data set, the results are quite surprising. This is comparing Mongo 3.0 with WireTiger and Snappy compression enabled, which is kind of the, you know, in terms of disk space, it's the same thing as MongoDB 3.2. Just didn't repeat the test, but it's the same thing. And on Green Plum, we were using Culinary Store and Compression 2 to make this test fair. So I don't know if you can read the bars over there, but it's basically this big bar is Mongo, this is small bar is Green Plum. And this is the same thing. These are kind of more or less the same, the index size. The table size is significantly, significantly smaller from 20 gigabytes to 70. That's more or less the difference. So very significant. And why is all this? Because we are storing the metadata separately from the data and we are not repeating the metadata all the time. And when you take that into account and you store it in a columnar way and then you compress it, it compresses very, very well. Because we have aligned data, we have classified the data, we have put it nicely on their own bins and now we have compressed those bins. So it compressed very well. What about the benchmark itself? So if we take some queries, and we're comparing here, if you remember, what we're trying to show here is to compare a query on MongoDB API versus a query on SQL once those tables have been generated by TorDB and important via replication, right? So if we want to obtain like, you know, like the distinct reviewers of products, the query in SQL looks like this, simple. Query MongoDB looks like this. Weird. But it basically does the same thing. So this is query one. This is another example of the query. Please let me know which one you can read and understand easier. The one on the left or the one on the right. Especially nice is this MongoDB allow this use true. This is because MongoDB has a limit of 16 megabytes return on an aggregate query. So if you have an aggregate query on MongoDB and that aggregation produces a result larger than 16 megabytes, like you're doing a string concat or whatever, you're out of luck. You need to spill to disk if that's bigger than 16 megabyte. And it's really been automatic. You have to explicitly say, please use the desk. Well, so this is the benchmark. I don't know if the ones on the back can see these tiny, tiny, tiny bars over there. Yeah, that's TorDB and Green Pump. Basically, first query that we're simulating took 969 seconds to execute to complete on MongoDB and 35 seconds to complete on Green Pump. That's a 28X improvement. The second query took 1,007 seconds on MongoDB and 13 seconds on Green Pump to execute. That's a 75X improvement. The third query took 31 seconds on Green Pump to execute. We don't know yet how much did it take on MongoDB because it crashes consistently, very consistently. We don't know the error. We don't understand it. So, you know, this is the idea. This is a real enabler for data warehousing on NoSQL. So, this is basically, please go to GitHub, download the source, try it, read about it. If you like it, please start it on GitHub. And of course, go and check out our FAC. There's many questions. Probably some of those are gonna be asked now, but the rest of them are gonna be there. So, just go and check them out. And that's it. Questions? Thank you. That's a good point. So, the way we have laid out the source code, it's very modular and has some abstraction layers. And one of those abstraction layers is the incoming protocol. So, that's one layer. And so, we right now have one layer called MongoWP, which is MongoDB wire protocol. Then it goes down to another layer called KiviDocument, which is an abstraction of a document. And we transform from MongoDB representation to an abstract key value document store. And then we process and do all the sub-document stuff and so on. So, it's not hard to speak another protocol and transform to KiviDocument. And then the rest of the thing will be exactly the same. So, it's not hard. Now, this is prepared for, I don't think we'll run well with pure key value stores. It will run better with document stores. Next on our list is gonna be couch base. More questions? I don't know, the order is pop up. Okay, here. I hope nobody, I mean, this is still on development phase. So, we're gonna release mid-February version 0.4. And the version after that is gonna be 0.8. We're going powers of two. And yeah, the first version was 0.1 and then 0.2. So, this one is gonna be 0.4, then it will be 0.8. And very soon after, it was gonna be 1.0. So, it's not for production ready yet, but I hope it will. And there's many people just trying out non-in-production loads. More questions? Over there? Okay. So, this is basically very, very straightforward. Because first of all, in NoSQL, in Unstructured Data, there's no type change. If you remember from what I was saying before, the metadata which specifies the type is associated with the document itself, comes with the data. So, it's just a different type. It's not that it changed, it's a different type. So, the way we process this is that if you remember, we just analyze the sub-document and find a matching data type and value. If there is no table for that combination, we will create a new table. In other words, we will create a new table. Now, next question. Isn't that a problem? Are you going to be created thousands of tables? Well, sometimes yes. Sometimes yes, that may happen. If there is a lot of kind of combinatorial explosion of different keys and values and so on, which sometimes happen, we'll end up creating a lot of tables. But fortunately, this is not a big problem, neither. We have, we did a previous R&D project some three years ago in Postgres where we created a billion tables inside a Postgres database. And it worked quite well. And table creation was very fast. We were doing 12k, 12,000 tables per second creation. So, you know, it's not a big deal. More questions over there? I'm sorry, can you repeat it again?  We just cut that key. There's nothing else we can do. Yeah. Yeah, there's some restrictions with identifiers. But, you know, we escape some values and we cut keys. Not really, as I mentioned before, yeah, React is mostly key value. So, we are kind of currently, we're targeting more document data source than key value source. Because definitely, a key value source is a completely different list. Even though some key value source is stored documents, which document is a value, right? But the other way around, it's not on our roadmap. I won't say, can't say it can't be done. Maybe it's great too, but I don't know. More questions? Sure. All right, anyway, I'm gonna be around. I'm gonna be also on the Postgres booth next two hours or so. Feel free to pop in if you want to, and of course, if you have any question about Postgres. And just let me know if you try it. Let us know, give us feedback, and hopefully you like it. Thank you.


And the ward congress in Ondo's state was relatively peaceful in most of the wards visited in Akura at the state capitol. The exercise was monitored by the Boiga Ishiaka lead committee and the three senatorial districts of the state. Addressing reporters, Ishiaka commended the turnout of party members. He urged any aggrieved members to direct the aggraviances to the appeal committee. It's encouraging and I think the party clearly is doing very well around here. We have not gone around to attest it but my members have gone to other parts of the state and from the report I'm also getting from them. Yes there are literally two scale missions here and there but that will now remove the fact that the exercise here has been good. The congress is on. People are participating. We are a member of one family. No issue at stake and across the two or three wards you go there you see a member in their various offices doing their world congresses. So whatever is happening here today is just like a carnival. It is one love. What we want to ensure is that the majority, the majority of the people takes the lead. The majority will have their way. Why the minority will have their say. Yeah so far looking at what is happening I'm from Akura south. My ward is ward five. I have been able to go around this camp and then there's absolute compliance to the regulation of the party. What we are doing right now is to make an affirmation to the instruction that the party had laid down. Now we know everybody today are very happy. There's no fight anywhere. Under you will hear from our leader. It's not in the word for. Under you see the school but in our chairman local government everybody happy for today. Hello. Hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


The literature review aims to identify gene polymorphisms associated with chronic periodontitis, CP, susceptibility, focusing on Illinois 1, Illinois 6, Illinois 10, vitamin D receptor and CD14 genes, but the evidence is limited by varying carriage rates of rare alleles among studies and underpowered designs. A more comprehensive insight into gene polymorphisms contribution to CP requires larger cohorts, well-defined phenotypes, control for other risk factors, and analysis of multiple genes and polymorphisms within the same pathway. This article was offered by Marger L. Lane, Bruno G. Luz, and W. Krylad.


Hi, thank you all so much for joining. My name is Seth Leesman. I'm a Solutions Architect for Verica and commit on Apache Flink and I'm really excited to be giving this hands-on demonstration of building state for serverless applications using Apache Flink stateful functions. So if you're already familiar with Apache Flink, you likely know it as a world-class stream processor. It's very popular in the data engineering space for continuous ETLs, real-time aggregations, and reporting. And so there's this obvious question, right? What does a stream processor have to say about serverless? And at the end of the session, I hope you walk away thinking quite a lot. So to begin, I want to do a little bit of table setting. I think these terms serverless and stateful serverless have become a bit loaded in recent years and that a lot of people are using them to mean many different things. To me, serverless is not simply these commercial function as a service products, although that certainly does fall under the umbrella. Really, it is a realization of modern infrastructure capabilities, allowing us to iterate more quickly and with more confidence. So if our business is running a web app and business is doing really well, traffic spikes, we need to go from one instance to three. We're no longer requisitioning hardware, installing VMs, setting up networks. Instead, we simply increase our replic accounts. Stateful serverless at its core is really just about bringing these advances to the application layer, along with some key primitives that any real world application needs. So consistent durable state. Your application needs to be able to retain information it can act on in the future. Cloud native fault tolerance. So as we are maintaining the state, we want to do so in a way that leverages what this underlying modern infrastructure is really good at. And to make our lives as easy as possible in production. And simple messaging primitives between systems. Your business is not built on a single application, but a whole host of systems that need to communicate with each other in arbitrary and complex ways. We want to make this as easy and intuitive as possible. So this diagram shows a very traditional two tiered application architecture, something I'm sure everyone here is very familiar with, where business logic is deployed via a stateless tier, giving us those nice serverless benefits. State is then managed on a separate tier via database or other data store details here, not so important. But when an application receives a request or other message, something to trigger computation, it will likely communicate with that data store and potentially update some values. And then do one of three things. One, it could do nothing. It could be that this message was simply to update state. We have done that and we are finished. Two, we may send a result back to an end user. So this message or this trigger was to read some information or querying something. Or three, we're going to invoke another service, repeating the cycle. You can think of these different components interacting with each other as a sort of data flow. And the obvious question arises, what happens when something fails? Well, a fundamental problem with this architecture is that for any failure in any call across a service boundary, it becomes very hard to reason about which of the desired outcomes were actually achieved. Applications are forced to have a method of determining whether or not to attempt to retry or somehow make their state updates item post. But what if we rethought this problem in the beginning? What if we inverted it so that messaging runs through the database? Well, it turns out this is exactly what stream processors like Apache Flank have been doing for years to provide what we call exactly one state semantics. Business logic remains stateless and is deployed as a separate service from data storage. But this time, messaging is going to flow through the database and the data store is going to invoke functions and stateless containers. Supplying state is the part of the payload of each message. These application functions will then return back both state updates and messages to be sent to other functions. By moving messaging from the compute layer to the storage layer, state in messaging is easily made atomic. And if messaging were to fail for whatever reason, the state update is also rolled back. So retries are always item components. This is exactly the approach taken by Apache Flank stateful functions. We are using a flink cluster for message routing and state management while allowing the actual functions containing application logic to be deployed in a separate compute tier. This gives us a very powerful runtime where compute and state are logically co-located for consistency, but at the same time physically separated. All state accesses and updates are integrated as part of the function invocation request in response. So our business logic can be deployed however we so choose. Could be standard Kubernetes service using orchestration tool like Knative or even a wholly managed service like AWS Lambda. Yet we are able to retain consistent state and messaging. So as we go through our checklist of a proper stateful serverless framework, the initial requirements under pure serverless are easily met by deploying our business logic in stateless containers separate from everything else. But what about the stateful specific requirements? Well, to understand that, let's talk about some of the core concepts in stateful functions. When developing an application, you're going to implement several services or what we call functions that are basically small pieces of code or logic representing entities within an application. You could, for example, define a function type representing a user with a single instance of that function type representing a single user within our application. Think of this as an object or in the terms as being that it's between a class and an instance. These function instances are invocable through messages and do not consume resources while inactive or simply when they're not being invoked. What this means is the runtime can host a theoretically infinite number of function instances within a fixed finite set of resources. And this whole thing is polygraph from the ground up. So we are deploying these functions in our own containers, meaning you can do so in any language you so choose. The only requirement is that the language supports HTTP, GRPC or UNIX sockets, which is to say we support virtually every language. Communication between the flint cluster and user code happens through a very well defined and small protocol. Certainly something you could develop against yourself. At the same time, we realize that most people don't want to do that day to day and so the community does ship a number of predefined SDKs that wrapped up protocol in higher level idiomatic constructs for that language. So there's an SDK for Python, active development on Golang and Rust. There's even a Haskell SDK that recently popped up in the community and hopefully your favorite language coming soon. Adding new SDKs is very high on our prioritizations. So we can have user code, we can write different languages, lots of people can do that. Where things get interesting is that we can run these functions with dynamic messaging and consistent states. So if you have used Apache Flint in the past and you're familiar with this idea of a Dataflow DAG, that is completely gone. Instead, we support arbitrary communication between functions using logical IDs. And so the only thing an instance needs to know to message some other function instance is its function type and ID. So what sort of function do I want a message in which particular instance? If we were maintaining user function to keep track of users of our business, we would have user as our function type and there would be an instance for myself. My ID would be someone else's might be John or Eagle or Gordon or whoever else. And all of this can be done with exactly one semantics. So function instances are able to maintain local state. While the runtime ensures that messaging and state updates are integrated so users can have out of the box efficient consistency. This is true across that inputs to the application, application state itself and outputs delivered from the application. And I think most importantly, all of this is no database required or better put, we are using Apache Flink as our database. So Flink has long provided large scale consistent state management through these concepts of state backends and distributed snapshot. State is stored locally within the cluster for fast accesses and is periodically backed up to simple blob storage. This could be Amazon S3, Google Cloud Storage, HDFS and NFS drive, MinIO, whatever you already have available. In the case of failure when a pod that is part of the Flint cluster itself restarts for whatever reason, it will simply download its latest snapshot and continue on processing. This means we are not reliant on stateful sets or persistent volumes for high availability of states. The only thing we need highly available in the system is our blob storage, which is the easiest thing to achieve. Using those model organizations of scale to managing hundreds of terabytes of state within Flint applications themselves with the confidence they're delivering consistent, reliable results. So that is enough on concepts. Let's take a look at some specific SDKs, actually build something. Today we're going to be looking at the Python SDK in particular, but all of these concepts translate to all of the different SDKs. They all offer the same core primitives. So we need to begin by thinking about types. Because remote functions can be implemented in any language and a single application can be composed of many functions written in many different languages, we need a uniform format for communication. And for that we've decided to standardize on ProtoBom. If you're not familiar, it is a serialization standard out of Google that has very strong cross-language support. And so all messages passed between functions must be encoded as ProtoBom. And in particular they must be encoded as ProtoBom any, which is very convenient because it contains what the logical type along with the serialized bytes. And so within a particular user function, you can then quote unquote unwrap that any message into a specific concrete type using your language specific ProtoBom library that you can then work against. Same thing goes for state type. So anything we want as consistent durable state must be ProtoBom any. And this allows state written in arbitrary languages to be uniformly maintained by the Flink cluster. Flink's state backends are simply going to store the serialized any record. At the same time, we realize this is kind of boilerplatey and so, and it is if you're working directly against the protocol. But for all of the language SDKs, we do offer higher level constructs so that you only ever have to develop against specific ProtoBom types. Using say the Python SDK, you will rarely if ever actually see in any record. So as with any good introduction to a new bit of software, we're going to start with Hello World, but make it state fun specific. And so we're building a greeter application that is going to greet users of our service based on the number of times that specific user has been seen so far. So every user is going to get a personalized greeting. First time I'm greeted it might say welcome stuff. Second time it may be welcome back stuff and third time third times the charm stuff. Yet if someone else is greeted they are going to get their own personalized greeting. And this is going to show off some very important primitives. So we're going to talk about messages, right? How do you a Greek request for myself specifically? How does that get to a function and state? So we need to maintain for every user a count of how many times they have been seen. So each function instance is associated with a function type and ID as I said before, which forms its unique address. This logical address is what we do use when messaging that function. So when I am to be greeted we're going to send a Greek request to the function type greeter and the ID set. We can see that as our input to the function. So this is the message that was being passed to us. Again as I mentioned while the runtime is using protobuf any by leveraging Python three types in this case we are able to have the SDK automatically unwrap that for us. Similarly we can send our result to another function. We'll look at the middle bit of creating the greeting in just a moment. But we are going to both pack our result into an any type so we can avoid that boiler plate and send it to another function instance. In this case we're sending it to an email sender that is going to ship out that greeting. When messaging we're using our address so we have our function type which one is it well it is email sender that's the sort of function I want to message. Which specific one well I want to message the email sender for this specific email address. And then we can go into our personalized greeting itself. And this is showing off what I think is our most powerful feature which is durable state. All this method is doing is keeping track of the number of times this particular user has been seen so far and then generating a message based on that count. So our state is being accessed via our context and we are able to read out our state based on some name and specify the type. So we're keeping track of scene count which is a portable type I've predefined. We can both read that type out and write it back. And you know what that's it the rest of this is standard Python there's nothing state fund specific about the rest of this method. The only thing that we have done differently than say building this in your CS 101 course is that our variables are being managed via the context instead of basic instance variables right so we're using our context but otherwise it's just Python right and we get all these nice primitives like durability out of the box. So we've written our code right but we have to make it available right it's it's running in some remote container. And so the first thing we need is our function registry. This is going to map logical function types to concrete bits of code. So in this example we have written both our greet function and our send email function. They're both written in Python and they're both written in the same file but neither of those requirements. The send email function for example could be a rust function or it could be implemented in Haskell and it could be running halfway around the world from our reader. But we're going to bind these for registry and we're giving the function type. So when we shoot off a message to that type we know how to associate it with a specific concrete method. And then we need to expose it to the flint cluster and we need to ensure that it actually works against our protocol. And for that we ship a request reply handler which dispatches the invocation requests to the bound functions and then encodes their side effects both resulting output messages along the state updates as an HTTP response to be sent back to the flint cluster. And then we simply expose this handler using your favorite HTTP framework in this example and the later examples I'm using flask but that is not a hard requirements that is just something that I chose to use. Plug in your favorite library here. Okay so readers are interesting readers are fun but that's not what you're building your business is not built on hello world applications but it might be built on model serving. So we're going to take a look at building a fraud detection application specifically for credit card transactions. So as a transaction comes in we want to build up feature vectors which require looking at states right we need to remember things about our users and our merchants. We need to query these functions in dynamic ways and then we want to score that against something that was likely provided by our data science team. Giving us back a score on whether or not we believe this transaction to be fraudulent and at that point we can take some action. Okay let's take a look at the code for this model survey example. Again we're going to be sticking with the Python SDK and for simplicity all the functions are implemented in a single file as a single flask application but just reiterate make it very clear that is not a hard requirement. Simply for simplicity of this demonstration these functions could all be implemented in different languages they can be packaged and deployed separately that is supported and expected workflow of many state fund systems. So we're going to be building up feature vectors whenever a transaction comes in we need to get information that we can use to send to our model and one of those features is a fraud count. So how many times over the last 30 days has this particular account reported and confirmed fraudulent activity? The idea here being that the more often that we see fraud for a particular count the more likely we are to continue to see it in the future. It's a rolling 30 day sum because people's behavior changes and so as things become further in the past they become less relevant. So our function type is ververica slash counter right this is the logical type we will use to message this function and we take into parameters our context it gives us access to capabilities like state and messaging and the actual message that was sent to us. Leveraging Python 3 type annotations we get to avoid all of our any protocol boilerplates and I'm using a union type here because we support working against multiple message types. So let's start with this confirmed fraud message. A record is going to come in say from a Kafka topic that tells us that a user has confirmed fraudulent activity against a particular account. This function I forgot to mention is always scoped to a particular account ID so fraud count is our function type. Account is going to be our ID for the logical address. When this comes in we need to increment our accounts and so all we're going to do is go into our context we're going to read out the current accounts and then we're going to increment it if it already exists or initialize it if there has been no fraud over the last 30 days for this particular account. Once we have done that we will simply repacket we will set that value and we're done. So while we have switched to using a context versus local variables we are otherwise just writing very simple Python code in getting fault tolerance and durability from the runtime. But I said we also need to do a rolling 30-day count. So every time I increment this fraud counts value in 30 days I need to decrement it. Well we're able to send messages to other functions but it turns out we're also able to send messages to ourselves. And more interestingly we can send messages with delay. So after we increment our count we are going to pack and send after and we are this means we're going to send a message where we're sending it well we're going to send it to ourselves. We have the context we can get the current address and so we're going to send ourselves an expire fraud message that tells us to decrement but we are going to give it a delay of 30 days. So this message will not arrive until 30 days after we send it. And the runtime is able to ensure this message is consistent and durable so that if we have failure over the course of that 30-day period for whatever reason this message will not be lost. And we do that and we're ready to go. So we see that expire fraud is also an accepted type and so after 30 days it will arrive and what are we going to do with it? Well we're simply going to decrement our value so I will read out our fraud counts. We'll decrement it and then if it's zero we'll go ahead and delete the state entirely just freeze up a bit of space and makes things more scalable. But this is really an optimization detail. Otherwise we are going to go ahead and set the new value. So if it was five it's now four we have decremented it and we are good to go. But storing state is fine. We also need to act upon it and so the third message type that this function accepts is query fraud. Someone can message a particular instance of this function, right, they can query for a particular account and ask how much fraud have you seen over the last 30 days. When we receive this we'll simply check our state value. If it's not already set if there is nothing there we'll give it some default and then we will reply. So send this message back to the caller. This is everything we need for distributed durable consistent state and messaging of this function. Let's not see how it's used. So I have some other functions in here we're going to skip past but the main function in this workflow is what I'm going to call the transaction manager. This is what coordinates the whole workflow and builds up our feature vector every time a transaction comes in. So again we have our context we have our message types the main one being a transaction so every time a user say swipes their credit card or does something else we will get a transaction event that contains the account ID it contains the merchant ID of where they were making this purchase and the amount of the transaction and so we see this we are going to cash it in state we want to hold on to this and make it available later on and then we're going to farm out to our different functions that we are using to build up our feature vector so you can see here we are querying that counter we just defined above and we are going to the instance for this particular account we're also getting some merchant information and some other values. When these functions respond right we saw that our fraud count replies back with a reported fraud well here it is when we get this what are we going to do well there is a bit of business logic to ensure that we have gotten all of our features if we haven't will store that reported fraud count in state until we get all different features back but when we have them all we are going to build up our feature vector and message our model this is likely living somewhere else it's provided by the data science team they're going to iterate and deploy the separately of the rest of the application and it will take in that feature vector computer score and respond back when it does so we are going to get this fraud score so this is our confidence interval from zero to 100 of how likely we think it is that something is fraudulent so zero being is absolutely not and 100 being this is absolutely fraud when we get that score we will compare it to some predefined interval and if it is above the threshold say 80 percent well we will send an alerts to a Kafka topic called alerts that says hey we think this is fraudulent and the user will see that and they can act upon they can maybe confirm it and the bank will block that transaction or they can say you know what this was really me please let it go through we're also going to delete all of our state values at the end because hey we have scored and alerted on this transaction we don't need to retain this information any longer as we have built all of these up we are making them available via the request reply handler and we're packaging this as a flask application I've defined an endpoint slash state fund that accepts a post and so whenever data arrives whenever we receive that we'll simply send the whole payload to the handler and it will manage dispatching to our functions encoding our effects our state updates and our responses and we will simply send that back to the caller of this endpoint which is the flink cluster when we go to package this so let's take a look at the dockerfile you'll see that there is nothing state fund specific here this is a plain and simple flask application there's nothing about the flink runtime we're going to see there's nothing special about this in any way and if we look at our dependencies as well we are including the state fund sdk which is what wraps that high-level protocol and then we are pulling in flask and whatever other python dependencies we need so where this the model function we might be pulling in numpy or scipy or any of those good data science libraries we have full flexibility here and when it comes time to deploy this well we are going to deploy it as a standard kubernetes deployment so i have written this deployment specification i've pushed my image i want 10 replicas of this because i want to be able to scale out we are exposing it under port 8000 but this is all stock and standard kubernetes additionally there is a service that is making it reachable so that gets us our user code but what about the flink cluster right how does it know where to all right so this file is our module that yaml file this is the configuration we give to the flink cluster that tells it how to map function types logical function types to addresses under which our functions are reachable so we can see here i have our counter function i have said that this is the logical function type so when you see a message that is targeting ver verica slash counter this is the metadata you should use uh the function is exposed as an HTTP endpoints and this is the specific endpoints you should use we also have at the bottom our ingresses and egresses this is how the functions communicate with the outside world so you saw for example that we were sending alerts to a kafka topic we're also reading our data from kafka topics uh let's look at the example of our confirm fraud uh message well i have said that this is coming from kafka uh i have given a name and i have my kafka specific configuration so where the brokers live uh consumer group ids things like that and then uh we give it a list of topics to consume from so we are reading from the confirmed topic we've specified our type url so what sort of data are we reading and then we give it a list of targets so what function types do we want to send these messages to we give it a list of types the id is implicitly pulled from the header and it will route our messages to the appropriate function to begin that computation along with kafka we support AWS kinesis out of the box and then if you're comfortable writing a little bit of java code we also support a whole host of other systems including JDBC elastic pulsar for vega rabbit mq and as we see demand we'll add more first-class yaml support for those other systems we we're going to take this file after we have written it and build our docker image so this base image flink state fund contains the entire Apache flink runtime along with all of these stateful function specific runtime code and all we need to do is copy our module yaml file on to the image there's no java code to write there is no flink specific code to write I am also including a flink comp which is some flink cluster configurations but this is stock and standard if you have written other Apache flink applications in the past and this is the image we are going to use to run our cluster and so I am in fact already doing that let's go ahead and take a look at our pods I have a Kubernetes cluster that is running three Kafka brokers for our data I have a data simulator that is simulating transactions and confirm fraud accounts and all those good things and then we are running our flink cluster and our user code so I'm running three instances three nodes in my flink cluster each of these only have a single core so it's very small and then we are running our user code and I have a replica set of 10 so I want to really scale out that compute if we go ahead and let's go ahead and believe I'm already port forwarded so I can pull over the flink UI you can see that everything is up and running this is the flink UI if you're not familiar it tells us what our application is doing and so we can see here we have processed since I started this roughly 200,000 messages let's these are all calling out to our user code it is being routed through this application it is syncing the results into Kafka we take a look at our checkpoints at our fault tolerance we can see that things are going through smoothly there was a failure but that's okay we handled it gracefully and currently I'm managing about 7 gigabytes of state within the flink cluster remember this is being stored locally either in local memory or spilling to local disk but it is always local we are never using persistent volumes of stateful sets minio is providing all of our fault tolerance and when it is time to go make a change so maybe I want to change my replica count or I want to deploy a new version of my user code all I need to do is apply those values that we have for our function so I can qctl apply-f statefund functions and this has our deployment YAML and our service YAML and this will apply those changes in this case I haven't actually changed anything we can also apply a horizontal load balancer so perhaps I did not want to have a static set of functions but I want to scale as my load goes up and down throughout the day we can do that and we will be able to do so gracefully and we can multiplex different function modules together so I'm running this code perhaps I'm more of a data engineering team and so we're in charge of building the feature vectors maintaining that state the data science team has their python code that is our model they're going to deploy that separately and make their own updates and we can all do that gracefully and consistent so I really appreciate you taking the time to listen to my talk today I hope you are excited about stateful functions and the future of stateful serverless applications if you have any questions I'm always on Twitter at SJ Weaseman also the Apache Flink user mailing list is the most active user mailing list of any Apache project and it's a great place to get help thank you so much and I hope you enjoy the rest of the conference


I came home from work on a Friday night ready to start my normal NBA season ritual. Drink 12 beers, watch the game, and crawl in bed a fat happy drunk. All that came to a grinding halt when I came home to find our power had gone out. All of our neighbors still had power, so our roommate obviously hadn't paid our power bill. I called my roommate who left that morning to go on vacation with his family and told him he needed to get this fixed ASAP. I wasn't planning on spending my weekend in a dark apartment with no NBA.  An hour after I called him, my roommate called me back saying the bill had been paid and we would get power back in probably a few hours. At that point I decided I'd go to a friend's place, watch the game, and by the time I got back home around midnight, the power would be back on. Fast forward and I'm leaving my friend's house. My phone had died an hour or so ago, so my friend called a taxi for me. I got dropped off and the taxi drove away. I looked up at the apartment building and saw all of the apartments were lit up with electric light. Some burned a bright yellow, some a moody blue from the TV. My apartment was the lone black window. There was still a chance I had power. Maybe I just didn't leave any lights on when I left for work that morning. I opened the apartment door and looked into the pitch blackness of the apartment before me. The closest light switch was about 10 feet from the front door, so I would need to feel along the walls for it in the blackness. The door thudded shut behind me as I felt my way along the walls, leaving me in near total darkness. It was much darker than I expected in the apartment. It didn't help. We usually kept the blinds closed to thwart nosy neighbors. I must have missed the light switch because the first thing I felt in my hands was a door frame. It was my bedroom door. I wandered in, still hugging the wall, and my door whined a loud creak as I leaned up against it. I hated that door. The slightest touch would cause the door to seemingly scream.  Eventually, I found the light switch. Nothing. Shit. No power. I kept the switch turned on, so if the power returned soon, I would know immediately. I remembered thinking camping. This is pretty much camping.  I would crawl in bed, and hopefully we'd have power in the morning. I wanted to brush my teeth first, so I felt along the walls and found my way into the bathroom. It seemed even darker and more oppressive than the rest of the apartment. Around the corner from any windows and having no windows itself, the bathroom was essentially a black hole. No light could escape it. My hands grasped around my cluttered bathroom counter, finding toothpaste and after some time, my toothbrush. I stood brushing my teeth, wishing for light, and listening to the sounds that accompanied living in a crowded apartment complex. I heard footfalls in the apartment above me. I could hear the neighbor couple next door arguing. And then I heard the unmistakable creaking of my bedroom door. The first thought I had was that the power was back on. The air conditioner must have turned on, and the air pressure moved the door and made it wine. Then I realized the light in my bedroom hadn't turned on. Someone was in the apartment. I suddenly felt exposed in the blackness of the bathroom, like I was being watched. I thought about the stories I heard of pirates wearing eye patches to adjust their eyes to seeing in the dark, to have the advantage in night combat. I hadn't heard the front door open, so if something was in here, it had been in here longer than I had and probably would have adjusted to the lack of light by now. I swallowed my toothpaste, too afraid to make noise, and I sat crouched in the blackness of the bathroom, ears straining to pick up any sound coming from my bedroom. I remembered I had a power bank sitting in my nightstand. If I could find that, I could charge my phone and then I'd have a flashlight. After five or so minutes of working up my courage, I slowly crept out of the bathroom. It seemed to have gotten even darker in the apartment in the last ten minutes, and I walked slowly and silently with my hands outstretched, probing the darkness. My hands were at chest height, groping in the direction of my bed. I found my lamp and worked down to the nightstand on which it sat. I opened the top drawer and started feeling for the brick like power bank. Every sound I made, every accidental click of my fingernails hitting the wood of the nightstand felt like I was making myself known to a predator. I felt like I was being hunted. I felt like a sitting duck. I found the power bank and the necessary cords and stood up. I didn't want to wait in my bedroom while it charged. I felt too exposed, so I planned to bring the brick and my phone into the bathroom again to wait. I stuffed the electronics in my pockets and started the process again of groping around in the blackness for the bathroom door frame. When my hands brushed up against something unfamiliar, did I get lost in my own bedroom? I reached back out with both hands. My left hand felt the right side of the bathroom door frame. I reached my right hand out to the wall to confirm and I felt it come down on that unfamiliar thing. My fingers tangled in a mess of a fine stringy material. I grabbed it and realized I was grabbing a full head of hair.  I didn't even think to scream. I bodied my way past the intruder and into the bathroom. I slammed the door shut behind me, locked it and stood in the blackness.   Either way, it had probably sat there watching me in silence as I was fumbling around in the nightstand. My heart was pounding out of my chest and I felt like I was going to vomit. I sat in silence, pressed up against the bathroom door, listening to the other side. I could hear the neighbors still, but I couldn't hear anything in my bedroom. Then there was a knock at the door. It was a polite knock, as if checking to see if the bathroom was occupied. Then it started getting louder. Heavier, more rapid. I thought whoever was on the other side would soon just punch straight through the cheap door. Then silence. I hated how long it took iPhones to recharge. I need light now. I remembered I kept a candle and matches on top of my toilet, so I felt my way over to the other side of the bathroom and found them. I sat for a while, listening and thinking. I kept thinking about the pirates and I didn't want to lose the progress my eyes had made adjusting to the darkness. Then I heard from the other side of the bathroom, water dripping. How had that just started dripping? It sounded like it was coming from my bathtub. Was something in here with me? How long had they been in here with me? Were they here when I was brushing my teeth? I felt like I was going to be sick. I had to use the candle. I lit the match and caught the wick of it. A dim yellow flame cast light in a few feet in all directions, but my bathroom was long and I still couldn't make out the far end near the bathtub. I slid the candle along the floor to the middle of the bathroom. In the dim golden light, I could see two dirty feet sticking out from behind the shower curtain. I raised my eyes up and I saw a face also poking out from behind the curtain. It was dark and I couldn't make out many features except the dark eyes reflecting the light of the candle and its huge smile. I don't think it was human. I was flooded with fear. Me and the thing stared at each other. I felt like a chipmunk staring down a lion, each daring the other to make the first move.  I heard a voice from the other side of the door whisper.  Whatcha gonna do? The voice behind the door asked a little louder. I saw a hand appear from behind the shower curtain, its fingers each five inches long, seemingly sharp at the ends and black with decay. It started to slowly pull back the shower curtain. Whatcha gonna do? The voice outside the door was screaming now. The monster behind my shower curtain slowly moved one long decaying leg towards me. I felt the vibration in my pocket, letting me know my phone was charged. After breaking eye contact with this monster, I reached in my pocket, turned the flashlight on and pointed it at the thing. It opened its mouth wide, revealing rows of sharpened teeth and it screamed like what sounded like dozens of men's women's and children's voices. I broke for it. I ran out of the bathroom, my phone's light illuminating the room enough to see that there wasn't just the person whose hair I touched, but several people and they were all smiling. I ran out of the bedroom, feeling hands reaching out to me from the darkness and pulling at my shirt. I reached the door and flung myself out into the hallway and ran down the stairs and into the parking lot. I'm sitting in my car now. About ten minutes ago, I saw my bedroom light turn on. Our power's back. I think I'm still gonna wait until morning before I go back in there.


This is a National Science Foundation Future of Manufacturing project that started almost two years ago. It involves Purdue University as the lead and partners at Harvard, Tuskegee University and Ivy Tech Community College. The main goal of the project is to bring artificial intelligence and the latest advances in data analytics to small and medium manufacturers. The subset of project that I'm focusing on is the utilization of current sensors or IoT devices, which is just Internet of Things. And essentially the goal of this is to optimize and observe the program that any system is running while not having to physically be there monitoring the system and the operations itself. I was offered about six different research opportunities, but being able to have a one-to-one connection with my professor, my mentors, graduate students here, honestly I feel like this was the best choice to do that my summer.