Databases, a database seminar series at Carnegie Mellon University is recorded in front of a live studio audience. Funding for this program is made possible by Ottotune. Google. We're excited today to have a talk from Alex Kangway, who's a researcher at VMware Research and he'll be talking about the Center DB. Alex did his PhD in Rutgers and computer science. He did his undergraduate in Rutgers and math and another math degree in Princeton. Saw the light, saw the databases were the way to go in life, and he switched over. So we're happy to talk. So as always, yeah, how it did not. So as always, as Alex is giving a talk, if you have any questions, please unmute yourself and fire away at any time with the conversation for Alex and not him talking to his screen in Zoom for an hour by himself. So Alex, we appreciate you being here, the floor is yours, go for it. Thank you. Thanks. I really appreciate it. I want to reiterate absolutely to interrupt me. I have a flexible length talk, so there's definitely stuff I can, that's totally optional. So if we eat up a lot of time talking about other stuff, that would be awesome because I think it's a better way for the talk to go anyway. All right. So today I'm going to talk about SplinterDB, which is a key value store that we've been building at VMware.  These are people from research and engineering, kind of all over the place, as well as individual contributors. So I'm going to start with the story of SplinterDB. And so this is kind of one way of telling my story. So SplinterDB starts with a product group called BCN at VMware and they needed a new way of storing metadata. I'll get more into that in just a second. And then it kind of goes away from the sort of systems into theory because the way that we decided to solve this problem was by modeling it as an external memory dictionary, which is a sort of theory problem. And we came up with a data structure that's inspired by some techniques that were used in external memory dictionaries, which we call the maps be the epsilon tree. And that data structure we implemented and turned into SplinterDB, which is sort of the end product. All right. So let's start with BCN. So I want to start with what is BCN? So BCN is a virtualized storage solution. So the idea is that you have a machine that's running some virtual machines and it has some local physical devices and you want to slice and dice up those physical devices into storage containers that the VMs can access. You know, maybe they have some SLAs or something like that. This is sort of the simplest version of what BCN does. Okay. So what is this metadata that is the problem? So in a virtual storage, kind of like in virtual memory, you have to maintain a collection of virtual to physical translations. So if I want to go access some data that's sitting in one of these virtual storage containers, I'm going to have some sort of virtual disk address. I need to translate that into the physical address where the data actually lives. So you can think of this as being sort of like a page table. But unlike a page table, it's a little bit less specialized. There's a bunch of other metadata that's stored as well. So you can't just rely on some narrow solution to solve this problem. You have to use something general purpose. And another difficulty here is because this is storage, all this stuff has to be durable. It's not something you can just keep in memory and forget about. Okay. So why is this particular problem hard? Well, there are several reasons. One is this is on the critical path of all IO. So it has to be really fast.  So B-SAN is supposed to be a nearly invisible part of the storage layer. So you can't consume a lot of resources in doing this.  You can't use a lot of memory. And you don't want to use a lot of CPU resources either, right? This is supposed to kind of just act like it's not even there. This problem really came up as B-SAN was trying to support fast new devices like Optane-based SSDs or other ultra low latency devices. Because on slower devices, you can kind of do a lot of stuff and amortize against the slow device. But on faster devices, those kind of solutions didn't work anymore. And the last problem I'll get to more in a second, but it's hard because metadata is a fine grain.  And so you have to be careful with how you read and write that metadata. And really at the end of the day, the problems that stay there are key value stores couldn't keep up with this workload and would always become the bottom layer. So we decided to create something new. All right. So at this point, I'm going to cross over into theory. I'm not going to talk about theory very much in this talk. This is really just a little bit of flavor of what we were thinking about and where we're coming from. This is really going to be a systems talk or really it's going to be a data structure talk, which is kind of like, you know, bridging the gap between theory and systems. But we'll go into theory for just a second. Okay. So how do we think about this problem? How do we model it? So I was just saying that metadata is fine grade, right? So each individual piece of metadata is really small. So here on the slide, I have them as being 48 bytes. That's not actually what we use in vSAN, but it's sort of in the ballpark. And you know, each IO to a device, even a really fast device like an Optane SSD or an Ultra-Low latency SSD, has to happen at four kilobyte granularity. So you have this sort of mismatch here. If you just go and write each individual item directly to the device, you're going to, it's going to be extremely expensive. You're going to have a huge write up application. And the same thing can happen on the read side. So this problem is really well modeled by what's called the external memory model. In the external memory model, you have an internal memory of size M and you have an external storage, which can be unlimited in size. And you can read and write to and from the external storage in blocks of size B. And when you go to analyze your algorithm or your data structure, each one of these blocks that you read or write counts as an IO, and the total number of IOs that your algorithm or data structure uses is the cost of the algorithm. So here B is going to be the number of items in an IO. So here are items of 48 bytes. So B is going to be roughly 48 kilobytes over 48 bytes. So it's like 100-ish. And something to note here is this model only works in this sort of setting. If you make the items larger, then it won't work as well, right? So if you had a kilobyte items or four kilobyte items or larger than four kilobyte items, this wouldn't really be an appropriate model. And something that's sort of interesting here is that this external memory model really came up initially as a way to model hard drives. Because in a hard drive, any sort of item that you read or write to the device is going to be much smaller than the sort of good performance, the granny lighting which you get good performance of hard drive. You want to be writing to a hard drive, reading and writing to a hard drive in megabytes, ideally, in order to get the best performance. And so if you have something that's several kilobytes, that's only a small fraction of one of these blocks. One of those sort of like good writing efficient IO blocks. But here, because our data is so small, the external memory model works well as well. So we're going to stay in theory for just a little bit longer. So that's the external memory part of this. So this is also a dictionary problem. So what dictionary is is sort of the theoretical version of a key value store. It's a data structure, an abstract data structure in which you can insert new items and you can do point reads of those items. Sometimes dictionaries also include deletions and sometimes they also include scans, but the simplest version is just insertions and point reads. And so what's sort of interesting about external memory dictionaries is they come into flavors. And those two flavors of dictionary have different performance limits. So in theory speak, they have different lower bounds. And we're going to see that in practice, these lower bounds make a big difference. OK, so the first flavor of dictionaries are comparison-based dictionaries. So in this model, you have items that are stored in your dictionary. You have keys stored in your dictionary and you're not allowed to do anything with the keys other than compare them to each other. So if I have two keys, I can only ask, is one less than equal to or greater than the other one? And this means they're not allowed to do things like using hashing, I can't use filters, or if you can think of something else clever to do with your keys, you're probably not allowed to do it. You just imagine them as these opaque things that you can compare against each other.  And what this says is that if you can perform insertions sufficiently fast, so this is lambda over b log base lambda of n, then your lookups are going to be somewhat slow. So there's this trade-off here between your insertion speed and your lookup speed. The actual formulas aren't really that important, right? But the point is that there's some trade-off where if you want to make your insertions faster, sure, you can go and do that, but your lookups are going to get slower as a result. Okay, so what if we drop this comparison part of this model?  So I have a key, I can hash it, I can use filters on it. The reason I'm presenting these two things are these are the most natural thing that you would want to do with a key other than compare it. But you can do anything else. So then there's a different lower bound called the Iacona-Petraskier lower bound. Now this lower bound looks almost exactly the same as the Brutal Foggerberg lower bound, but you get to drop this log base lambda of n part in the insertions.  So what this says is if you can use hashing or filters in some clever way, or something else that's not just comparisons, you can get this basically free insertion performance boost over the comparison base dictionary. And in practice, this log n term is pretty big. It's like a factor of 10. So it's something that you actually care about. This isn't nothing. Okay, so there is two lower bounds. And we have a question from Ben. Yeah, when you can hash your keys, why is lookup none of one operation? So that's an interesting question. So you can do that. So for example, suppose I have a hash table, just like a regular in-memory hash table. But I put it on disk. You can get O of 1 reads, but then you're going to have really slow writes. The problem is each one of those writes is going to have to access a block on disk and perform a write. And you might be like, oh, well, that's only one. That's not a big deal. But the idea here is that in this model, this lambda over b term is subconstant in general, and usually quite subconstant. So that's actually saying, yeah, this is sort of fine accounted for by the model. I'm not sure if this is exactly in this trade off curve or if there's a slightly separate case for this. You can get your lookups to be constant time, but then your insertions are also going to be constant time, but you don't want. So that's a great question though. Okay, so in the, sorry. So in the literature, what's known about these types of problems are that there are optimal data structures for both of these lower bounds. So for this comparison based model, there are B trees and B epsilon trees. So B trees are optimal in a certain specific parameter setting, where reads are really sort of the only thing you care about. So they have sort of slow writes and they have basically reads as fast as you can get. BD epsilon trees are another data structure that allows you to get much, much faster writes at the cost of slightly slower reads. And you might sort of be like, well, I don't know about these data structure. These are some weird theory things. Well, so the data structure that comes up more in systems is the log structured merge tree, which is sort of very related to a BD epsilon tree. And while it's not actually optimal on this lower bound, if you just make some small tweaks to it that matter, sort of maybe more in theory than they do in practice, then you get something that looks a lot like a BD epsilon tree. And it would be optimal under this lower bound. Okay, on the right hand side, on the other hand, where you get these faster insertions basically for free, the things that exist so far are the icon of Petrascu hash table and something called a bundle of arrays or bundle of trees hash table. I just want to call this out because this is some work that I did in theory several years back. So this appeared at ICALP in 2018. But these things don't really work super well in practice, in part because they don't support scans. And also these are actually just kind of complicated data structures. Like, don't let the name hash table fool you here. These are actually kind of theory-only data structures. So the data structure that we use for Splinter is actually going to work on this right-hand side where we get faster insertions for free. And it's also going to support scans. And it does this by sort of borrowing some of the ideas from this earlier work of bundle of arrays, bundle of trees hash tables, and integrating them into sort of normal B trees, which are nice data structures and things that you can actually implement. Okay, so the rest of the talk is going to be basically what does this map BD epsilon tree look like and why does it have good performance? And how does it actually get implemented in SplinterDB as well as some other cool things we get to do with it? Okay, before I dive right into that, I'm going to give you some top-line performance numbers for Splinter because just because something is optimal in theory doesn't mean that it's actually good in practice, right? It's not unusual for theoretical data structures to kind of not implement so well. So first I'm going to introduce ROXDB because that's the state of the art that we compare against. So ROXDB is a high-performance key value store from Facebook. It's designed for solid state devices. Been around for a long time. They have a full-time engineering team that works on it. I'm sure you guys all know ROXDB. Okay, so here's the basic operational performance of SplinterDB against ROX. So here in this plot, we have a machine with 28 cores. It's got an Intel Optane SSD. The dataset here is relatively small key value pairs. So 24-byte keys and 100-byte values. And there's an 80-gigabyte dataset where the system is giving 25 gigabytes of RAM to the key value stores. So here on the left are insertions. So in this plot, the y-axis is the throughput and thousands of operations per second. This is a uniform insertion workload from YCSB. As you can see, SplinterDB is nine times faster than ROX. And it doesn't come at the cost of reads. On point lookups, this is YCSB run C. This is if you're in distributed reads. SplinterDB is two and a half times faster than ROX. If we look at the entire YCSB application benchmark suite, you can see that SplinterDB beats ROXDB on every single one of these workloads. All right, so how does this, where does all this performance come from? So it comes from the data structures that we use and like I said, this is gonna be what the rest of us are talking about. So the way I'm gonna get to the mapped BD epsilon trees, I'm gonna start sort of slow at the beginning and work up. So I'm gonna start with B trees, work through different variations of BD epsilon trees to get to what the mapped BD epsilon tree actually does. I start with a B tree. So a B tree is a B area search tree.  So if I wanna insert an item into a B tree, what I do is I start at the root, I compare my items key to the pivots in the root. This tells me which child to go to. I repeat this process in each internal node and eventually I find the leaf and I insert my item into that thing. I'm sort of well known that the insertion and lookup cost are logarithmic in the number of items that are in the B tree. Okay, so now that's the B tree. Let's go to BD epsilon trees. So BD epsilon tree is a search tree just like a B tree. But in internal nodes of a BD epsilon tree, rather than being packed full of pivots, have only a small fraction of the node have pivots and the rest of the node is used as a buffer. The leaves of the B tree, BD epsilon tree are just like a B tree. So there's packed full of key value pairs. And so something I should mention, I don't wanna gloss over is that the nodes in a BD epsilon tree tend to be much larger than the nodes of a B tree. So where a B tree nodes in practice, usually are four, eight, 60, maybe 64 kilobytes in the largest case. In a BD epsilon tree, they're often larger. A megabyte is not uncommon or more than megabytes is common. But you can kind of think of each one of these nodes as like an SS table in the log structure merge tree if you're more familiar with those. You play any tricks of having like the upper nodes have a larger size or smaller size than the lower nodes? It's an interesting question. So, in theory, you definitely can't. In practice would be interesting to see what you can do. Why can't you do that in theory? Well, you can do it in theory, but you don't get anything. This is already like optimal in theory. So like additional benefit you get in practice. Yeah, we gotta pay the bills, right? You gotta build the thing. So, I don't feel like theory's that easy. Constance matter, right? Yeah, absolutely, it does matter. That's interesting. I'm not sure exactly what would happen there, like whether you would get some sort of benefit or not. I'm just curious. Good question. Good question. It's kind of like a monkey flavor thing. Maybe you make your filters more or less, you know, large at different levels and you get something. I'd be interested to see if there's something there. Okay, so that's BD epsilon tree. So let's see how insertions work in a BD epsilon tree. So if I have an item I wanna insert, what I do is I just put it into the buffer of the root. And that's it. If this insertion is done, you don't have to do anything else. Once it gets put in the root buffer, you're done. So suppose I wanna do some more insertions, I'm just gonna keep putting in the root buffer and everything's going great until eventually the root buffer fills up. So what you do when a buffer fills up is you pick the child that's receiving the most messages, which in this case is this middle child, and you're moving to that child's buffer. And this is called a flush. So the thing to note here is that when you perform one of these flushes, we're always moving multiple items. And if you imagine these nodes hold lots and lots of items, not just four so they fit on the slide, then you're moving lots of items at once from node to node. So you amortize the cost of flushing across all of those items. Let's keep inserting some items in the root. We've made some room so we can do that. When it fills up, we're gonna do the same thing. We're gonna flush to a child. Now in this case, this child is also full. So we're gonna repeat this process. We'll pick its child that's receiving the most items and we'll move them to that child's buffer. And these flushes can kind of continue down the tree. So be the epsilon trees have much faster right performance than be trees because we always are moving multiple items down the tree at once. And so this batching gives us much better performance. If we imagine that the root node is always in memory, then at each point when we actually access the disk, we're doing multiple writes at once and we can even get subconstant write speeds so we can write cost. Okay, so that's how insertions work. What about lookups? Well, they're gonna work just like they work in be trees really, except now we have these buffers along the way. So we have to check in those buffers as well. So if I'm doing a point query for say 71, I'm gonna start in root. I'm gonna look at its buffer. If I don't find the 71, I'm gonna look at the pivots. It's gonna tell me which child to go to. I'll look in that child's buffer. I'll work my way down the tree. In this case, we ended up finding 71 in the leaf, but we could have found it a buffer along the way. Okay, so that's how lookups work. They're slightly slower than in be trees because we have these buffers along the way. So our fan out isn't as high as it would be in a be tree, but other than that, it's very similar. Okay, so I just told you that be trees have fast insertions and now I'm gonna tell you that they're not as fast as they could be. They're more expensive than they look. And what I'm gonna say here is also true of most loft or should merge trees. So why is that? So suppose we're doing insertions into a be depth cilantro and we have a node here and we're flushing some data from its parent into the node. Well, what do we have to do to make that flush happen? We have to read that node from storage and then we have to merge the data, new data into the data that's in that node. So this requires using the CPU and then we have to write the node back out to storage. So what I want you to notice here is that the amount of work we do isn't proportional to just the new data that we're adding. We also have to do work on the old data. So when we read the node, we're reading the old data. When we write the node, we're writing the old and the new data out together. And if we're merging the data together, that also involves having the CPU run through all the data that's in the node, both the old data and the new data. If we keep flushing data into this node, the data that's there, that's already there is gonna be written over and over again. So here in this sort of animation, the key value pairs that are turning darker red are getting written over and over again. And it turns out that this can happen up to be the epsilon times per node on average. And so this is a be the epsilon times blow up in the amount of write amplification that we have in the system, and also the amount of processor time that we're using up in terms of the processor being, doing all of these sort of compactions in these nodes. Okay, so we're gonna use new data structure that tries to address this problem called the size tiered be the epsilon tree. And this is work that appeared at ATC in 2020. So the original data structure that SplinterDB was built on. So a size tiered be the epsilon tree, it's just like a be the epsilon tree except we're gonna store the buffer discontinuously. So here in this picture, I have a be the epsilon tree node, it's got its pivots and the rest of the node is buffer. Now we're gonna break this buffer off from the pivots and we're gonna break it into several discontinuous pieces. So I'm gonna introduce some terminology that we use in Splinter to describe this data structure I'm gonna use in the rest of the top. So we refer to this pivot part of the node as the trunk node and the collection of all these sort of trunk nodes together in the superstructure as the trunk tree. And the trunk node has the pivots it also has pointers to these buffers and it has some additional metadata about the node as well. These buffers I'm gonna refer to as branches. So you have a tree and it's got some branches coming off of it. That's sort of the implicit metaphor here. So how is this gonna help us on insertions? So I'm only gonna show you one little piece of the tree rather than the whole thing because it gets very complicated. So here what I have in this slide is I have a trunk node, it has no data in it at all. So it just has some pivots, some metadata just that part of the node. And I'm gonna flush some data into it from its parent. So what you do in a size to be the epsilon tree is you just add that data as a new branch using essentially a pointer swing. So you add a pointer in the trunk node that points to this new branch that you've added. When it's time to flush some more data into the node we do the same thing. We add another pointer to the new data. And the point here is that we're not touching anything in that old branch. We don't have to read it from disk. We don't have to look at anything that's in there at all. So this can complicate things a little bit because these branches may have overlapping key ranges. So if I color in these pivots and the children that they point to and I color in the key value pairs in these branches you can see each branch can have keys that are going to any of the children. So they're all kind of can be mixed up in any combination you like. Okay, so let's add one more branch to this node. So here this again is just a flush from the parent adding a branch. And now let's say the node is full which really just means that there's some threshold for the amount of data that's in the node been passed. What are we gonna do? Well, we do the same thing that we do in a bdfs long tree. So recall what that means is we pick the child that's receiving the most items in this node which in this case is this orange child and we flush them to that child which means we read through each of these little buffers pull out all the keys that are going to this child and we merge them together and write them out as a new buffer they'll get added to that child as a pointer swing. And the key observation here is that each key value pair is read and written only once in a trunk node as it descends down the tree. So we don't have this problem of reading and rereading and writing and rewriting the data at each node. So this gives us a speed up over bdfs long trees. So your question? Yeah. So this must so lookups would take time with this, right? Because these pieces are that are not the trunk are spread across the disk and have pointers to each other and they're not also sorted. Yep, you're exactly right. This is right where I'm going. So as you pointed out correctly, when we do lookups we're going to have to check every branch along the path now instead of just the single buffer that was in node. So I'll just walk through this slowly but you're entirely right. We're gonna be checking in more places. So we do a query. If we look in a trunk node here I've squished down these branches so that they fit on the slide each one of these sort of blueish purple rectangles represents a branch key value pairs. We have to go look in every single branch down the root sleeve path. So we can end up looking a lot of branches as we go and we can eventually find our key value pair. This sort of overall other than the same but now we might have to look in more places. So if we actually look at the lookup cost as you're saying we get a larger lookup cost it turns out to be about bdfs long times more. So this makes us very sad. Another one of those. Yeah, go ahead. So what I don't understand is what is now happening is every, so when there is like an input it like it without any sorting without putting it into the final sorted array it's just placed in some empty area in the disk which is similar to what LSM tree does. I think my question is why are insertions insertions should be the same with the size tierd B epsilon tree and LSM tree. I can understand why lookups take time. So there are two flavors of LSM trees, right? There's leveled LSM trees and size tiered LSMs. And this is going to be roughly the same as a size tiered LSM on insertions. So where size tiered LSMs give you an improvement over leveled LSMs on insertion performance this is going to be the same kind of performance improvement here. So you can make that connection. There's like, they're not entirely unrelated data structures. That makes sense. So in the numbers that you showed in like in one of the earlier slides the LSM rock CB insertion performance was much less than splintered B performance. So that's rocks to be at least by default uses a leveled LSMs. That's where a lot of the insertion performance comes up. So where this is going is going to be how do you this is how you make insertions fast. And now the question is, how do you make reads? How do you fix the read the slow reads that are going to come up from this? So that's it. Yeah, no problem. I see some more questions. Yeah, leave go for it. I wish a leaf is also a former Tokyo tech. Yeah. Oh, cool. The factory pin. Yeah, I recognize some on the earlier slide with people on it. That's pretty cool to see. Yeah, I bet you've met like Doni and all those people, right? Very cool. Yeah. So I was wondering on your previous slide for the size to be epsilon trees after a flush what do you do with the elements that you flushed from the higher node? These empty spaces. You see them kind of disappearing, but like are you rewriting those buffers in place? Are you coalescing things in the internal node that they got flushed out of? What happens there? She keeps going down. That's a great question. So I was kind of intentionally leaving this as a mystery because I don't want to get too much of the details. At the end of my talk, we're going to talk about, I'm going to talk about a policy called flush that impact where I kind of explain this. But I think we might not get to that, which is fine. The idea is that we just mask them out and we kind of leave them there. And if there's enough, if they're like large enough that we can free up the spaces in those buffers for reuse then we go ahead and do that. If they're not, then we kind of just leave holes in the buffers. So these buffers here, I know I drew them small here but they're actually pretty big there. So they measured megabytes so we can, we're happy to cut holes in them and just use metadata to mask out the stuff that's been removed. Okay, cool, thanks. Thanks a lot. I think I'm all out of question as well. Cool. Yeah. Sorry, I might have missed something here. So we're still in external memory model, right? So you're pumping everything at blocks. Sure. You just mentioned that is buffers are megabytes size. So I got confused again. Yeah, so, you know, the way I would say this is we're not sticking completely rigorously to the external memory model. So, you know, on the kind of devices that we're really targeting, you know, for the external memory model, you'd expect the block size to be like four kilobytes or, you know, not much larger than that. And here I'm making these things larger. And it turns out that works better in practice for various reasons like, you know, a lot of times there are other parts of the storage system that become bottlenecks. So for example, going through the kernel interfaces in order to do writes and things, especially we use asynchronous, you know, support from the kernel and those things are slow. So it makes sense to have larger structures to read and write. We actually perform IO at like 128 kilobytes on the right path as much as possible and like four kilobytes on the read path. But this is getting into implementation stuff like that I don't think is super important, but yeah, we're sort of bending our relationship to the external memory model here a little bit to make things work out. And also because we want to cut into these buffers like this, right? You know, we don't want those sort of cuts to be at the sort of external memory size, not the whole buffers themselves. Because we want to be able to efficiently read out chunks of them. So I think that's a great question and, you know, yeah. Thanks. No problem. Anyone else? I think that's all I saw in terms of raised hands, but I can't see everyone on my screen. So. Yeah, you're gonna go for it. There's another question about bloom filter optimizations like LSM trees, if there is anything and I believe you have something of that later in the talk. Yeah, we're going to be talking about bloom filters a bunch soon. So let's go ahead and get to that. Okay. My machine is slowing down for some reason. I can't run zoom and keynote at the same time. Okay. All right, so let's start with how do you fix look up? So we're going to fix look up in two ways. One is going to be a sort of simple way that does a lot but doesn't work in all cases. And then we're going into a sort of slightly more involved way that really handles everything. All the things you can throw at it. Okay. So the problem here is that each node has multiple branches. And when we do point reads, we have to go look at each branch. And the first idea you could come up with to fix this is using filters to avoid looking at every branch. So filter is just a probabilistic data structure which enters membership queries with no false negatives. Somebody already mentioned bloom filters. There's also other variants like cuckoo filter and quotient filters. They all kind of do the same thing. So it's better to be we originally use quotient filters. So here in a trunk node, we could put in a filter for each branch. And I use these pink sips here to represent the filters. And now when we do a look up in the node, we'll only search those branches which contain the key plus some rare false positives that might come up. So if I do a query, I'll look in the node, I'll look in the first filter and say, now you don't have to look in this branch, look in the second filter, so now you don't have to look in this one either. I look in the third one says, yeah, you do. I go and find the key and this mostly works out. And it turns out if you can tune your false positive and rate small enough, then you can get lookups down to a very small number of IOs across the entire system. You can also do this in a single IO, if you barely more than a single IO, if you have enough memory to keep all these filters. Okay, so that's a simple thing. You said it mostly works out, when doesn't it work out? All right, thank you. So let's get down, really fixing lookups in size tier to be the epsilon trees. So the problem with this solution is you create all these filters that you have to look at and it can get really expensive. We have multiple filters per node and in practice on not even particularly large data sets, we can see 15 to 40 filter lookups per point query. And you might be like, well, it's not a big deal, you're probably gonna do an IO anyway, right? So why do you care about all these filter lookups? But there are a bunch of situations where that math doesn't quite work out. So the first is sort of simple, if your data set fits in memory or the similar situation where you are just, you have a few queries that you're doing a lot that fit in cache, you're not really gonna be doing much or any IO. And so this CPU limitation, the amount of CPU work that you do on all these filter queries is gonna limit your performance by a lot. Even if you're in the kind of like normal scenario where you're doing an IO for every query, the CPU costs all these filter lookups, it's still gonna matter because on fast devices, doing 40 filter lookups is not, is a substantial chunk of doing a single IO. So you're gonna need more threads to be able to drive the same query throughput to the device. And finally, in a case that's really important to us with our motivating example from VSAN, if you don't have enough memory to keep all these filters in memory, you're gonna degrade your performance because they're gonna get paged out to disk. And then you have this sort of bad choice. Well, you can go ahead and look at the filter that's on disk, but then you're doing an IO just to determine whether or not you should do an IO to actually go read it from the branch itself and the buffer itself. Or you can just skip the filter, but then you're just stuck reading all these branches again.  Okay, so this leads us to the solution we came up with, which is called Maplets. So what's a Maplet? A Maplet is a data structure that's like a filter, but it also stores small values. So a regular filter just answers the basic question. I have a key X and I can ask, is X in the set? The filter will say yes or no, and that's all it can tell you. Now a Maplet does a little bit more. It'll say, is X in the set? You'll ask, is X in the set? It'll say it's not in the set or maybe is in the set with some small value that you get, say maybe four. And you can even return a set of values. So it might say, yes, it's in the set with values three, four and seven. And that's sort of totally okay. Just like a filter, it has no false negatives and has the same false positive guarantee that a filter has. And it also turns out that if you consider the cumulative false positive rate of a collection of filters, replacing them with a single Maplet, even though it has to store these additional values, it'll have the same memory footprint as those multiple filters for the same cumulative false positive rate. The performance of Maplets, when you implement them using quotient filters, which is what we do. They have the same performance cost as a single quotient filter. So every lookup into the Maplet, if it's in memory is two cache line misses and you can perform lookups into them on storage at the cost of a single IO. Okay, so that's a Maplet. How do we use them in SplinterDB? This is where we get to map to be the Epsilon trees. So what we do is we replace these individual filters in a size to be the Epsilon tree with a single Maplet. So here I have these three pink sieves. I'm gonna change them into a pink signpost to represent that we've replaced them with a Maplet. And then when you use the values in the Maplet to store which buffers contain matching keys. So we're gonna use the buffer in this, we're gonna take our buffers or our branches, we're gonna order them and say, well, this is branch zero, one, two, three, and we'll store those numbers in the Maplet. So now when we do a query to the node, we just look in this single Maplet, it outputs us an index that we go look into and we immediately find, we immediately know which branches we need to look at. We only do a single query per node. So if this trunk node or this Maplet is in memory, this is fine, if it gets written out to disk, we can also access the Maplet directly from disk in a single IO. And so that one IO will tell us which of these branches to go to. So we never do more than one IO in a node unless we're actually going to a branch and we might do two. All right, so how does this actually play out in practice? I'll show you some read benchmarks we did with SplinterDB. So like I said, there are these sort of three scenarios where regular filters can fall short, low memory, medium memory, and high memory or hot queries. Let's start in the low memory case here. So what we predict is that in the low memory case filters, we'll get paged out to storage and so we'll have a lower look at performance overall. So here's a plot of what happens. So in this plot, I have SplinterDB in pink. It's going to be with Maplets in pink, SplinterDB with quotient filters in purple and RoxyB is here in blue as a sort of baseline. And here the y-axis is the throughput in thousands of operations per second. This is YCSBC, that workload. And on the x-axis, we have the number of threads that we're performing these queries. These are all synchronous queries. So we're not doing anything asynchronous or anything like that. Okay, so if we compare SplinterDB with Maplets to SplinterDB with quotient filters, they both relatively quickly hit a wall as to how much from how, you know, it doesn't require a lot of threads to hit the wall of how many queries you can do. And this is sort of because they're both maxing the device IOPS at that point. But SplinterDB with Maplets, because it can use these Maplets, can do IOs to these Maplets in order to, rather than having to do IO to all the individual filters in the nodes, it can get a bunch more throughput. It gets about 25% more throughput than SplinterDB with quotient filters. So if we move to the medium memory setting, so here we kind of predict, well, we're going to do roughly one IO per query because the filters fit in memory, but the data doesn't. You know, so what we predict here is that the cost of filter lookups means that we're going to need more threads in order to drive the same, in order to have the same throughput because the filter lookups are chewing up a bunch of CPU time on each one of these threads. And so if I show a similar plot here, notice the y-axis is at a different scale, so everything's way faster. It's not surprising because a lot more stuff to fit in RAM. You see SplinterDB in pink, and so with Maplets in pink and with quotient filters in purple, we kind of get to near the same endpoint once you have 28 threads, but with Maplets, we can get there much faster. We only need 16 threads to match the performance of SplinterDB with quotient filters at 28 threads. All right, so finally, we can look at the case where everything fits in RAM, or you know, you can imagine, like I said, that you have a few hot keys that you are acquiring. So in this case, we predict that you're basically not doing any IO, and the performance should be limited by CPU. And so unsurprising in this case, this is where Maplets really shine because we're doing, because all these, you know, filter queries and Maplet queries, all the filter queries are really expensive if you don't have any IO to amortize against. So here throughout, Maplets are about 50 to 60%, similar with Maplets of 50 to 60% higher throughput, no matter how many threads you use. And so unsurprising in this plot here that SplinterDB quotient filters is slower than ROXDB because ROXDB is level, so it doesn't have this additional sort of fragmentation that you have when you're sized here. With Maplets, we can get around that. Okay, any questions at this point? Should I keep going? Going to a slightly new topic. So I'll take this moment to drink some tea anyway. I guess my question is like, I mean, the sort of the Maplet and then the tear part, that's novel. I guess I'm, I mean, leaves here, we can ask them. How does this differentiate between, what was the key difference between this and the practical trees? So the size tiering is part of that.  And it is size tiering and Maplets are the main differences. Yeah, so it's like a data structural difference. I guess what would be interesting to see is the comparison between like what the, like between ROXDB, that's right, that's between SplitterDB and it's, it's map being Epsilon tree, sorry, the dog, and then the, and like the fractal tree from Toku Tech come back in the day.   Like, I don't know, do you want to say that that's, you know, incremental or something like that?  No, no, no, no, very clearly I'm not claiming that at all. I'm just trying to like, I guess I need to go read the fractal tree paper stuff again. I don't remember what their key demonstrator was. What they do is they use basically the BDF Epsilon tree, right?  And so innovation is going on size tiering and using Maplets and things like that. Okay, okay. Chris, Chris has a question in the chat. If you want to meet and start to go forward. Yeah, yeah, I was just, you had mentioned async APIs and I was just curious if you had any estimates of how well you benchmark with the async APIs versus sync APIs and SplinterDB. For reads? Yes, for reads. Sorry. So we have an asynchronous read operation. I don't think we've run these particular benchmarks. Basically what you get is you can max out the device IOPS with very few threads generally, but I don't have specific numbers to show you on that. Sure. But yeah, we do have that. Cool, thanks. No worries. All right, so Andy, should I also like, when should I stop this at like 50 minutes or an hour? An hour, so like, yeah, basically 10 minutes. Okay, great, I might be able to get through everything then. We'll see. Okay, so once you have maplets, you can do some additional cool stuff with them. And so one of those things I'm gonna show you is using maplets to manage space. So what do I mean? So one of the problems that people have with size tier data structures is that they can lead to redundant data which can waste space. So what do I mean by redundant data? So here, if I look at this node, because it has these different branches, I can have the same key show up in multiple branches. So for example here, key 41 shows up three times. So this first one here at the bottom with value one might have been an insertion and then its value got updated in the second branch and then it gets updated again in the third branch. You know, the same thing can happen with other keys. So like 79 appears twice and some keys might only appear once. And this can be because of updates. It can also be because of deletions and reinsertions. It's a thing that can happen. And so size tier data structure could waste more space than level tier data structures.  And compaction is gonna recover this disk space when there are many updates or many deletions by merging all these together. So we started with 14 items in this node on the left and we ended up with eight items on the right because we've merged all these items together. So why don't we just do that all the time? Just compact everything. Well, if we care about space, that might not help because suppose I have a node like this one here, it doesn't have any redundant data.   So where do maplets come in? So maplets can tell us how much redundant data there is in a node. How does that work? Well, if I look at this node on the left here and I peek at its maplet, I sort of open up the box a little bit. What does the maplet look like under the hood? Well, it's sort of a map from keys to the branch indices where those keys are, right? Now this is sort of a high level oversimplification. The maplet doesn't actually contain the whole keys. It contains sort of fingerprints of the keys or something like that. It's not really important, but you have some sort of representation of the key that maps to one or more indices. And what we'll notice here on the left is there are lots of multiple entries for these keys because these keys appear in multiple branches. So we can just look at a sample of these keys and see how many entries they have on average. If we do the same thing on the right, this maplet's gonna look very different. All these keys are just going to have single entries. And so if we see few multiple entries when we sample the maplet, then we know that there is not a redundant data in the node. We can put this all together into an adaptive space reclamation policy. So what SplinterDB does is it maintains a heap of all the trunk nodes, which is sorted by the estimated amount of redundant data that we get from looking at their maplets. So what we can do is every time we update a maplet, every time we rebuild a maplet because we flushed new data into the node, we update the estimate of the amount of redundant data. So we get this heap. And now we can have a policy that uses this. So for example, whenever this usage gets too high, we can initiate a compaction on the top node of this heap. So the top node of this heap will have the most redundant data. So that compaction will return the most amount of space, the greatest bang for its buck. And so the idea here is that we always are doing the most efficient compactions in order to recover space and we're not doing a lot of compactions that don't actually help us recover any space. So if we look at what happens, if we run a benchmark on this, so what this benchmark is showing, we have the same sort of set up as before. And here we're doing 100% uniformly distributed updates to an existing 80 gigabyte dataset. So here what I have in this plot is on the y-axis I have the throughput and updates per second. And the x-axis is the space efficiency of the system. So what space efficiency means is it's the logical amount of data in the system over the actual amount of space that the system is using. So here 100% means that you're not using any extra space at all. And 50% means that you're using twice the space as the data itself. So here we have in purple, we have regular SplinterDB. In blue, we have ROXDB. And in pink, we have SplinterDB using maplets. So these different labels on the SplinterDB nodes are the thresholds at which we start performing additional compactions. So for example, at 120 gigabyte threshold, that means once this system is using 128 gigabytes of compactions, 128 gigabytes of space, we're gonna start triggering these additional space reclamation compactions. So as you can see, so I should say that in this plot, space efficiency gets better as you go to the right. So the system is smaller to the right and it's faster as you go from bottom to top. So I just wanna point out a few things. So Splinter with maplets is always faster than ROXDB and it always uses less space, which is great. But also there's this neat thing as we start setting our space thresholds lower and lower, we do in fact use our space more efficiently and we sort of smoothly get the trade off the update throughput as we go. So that's kind of nice. So you can just sort of say, ah, this is my space budget and I would like to go as fast as I can with that space budget. And that's what adaptive reclamation gives you. All right, so I'm gonna try and do my last topic here. This is sort of the optional one that looks like I have about five minutes. So we'll see if we can get through it. So this is just a cool optimization that we did in SplinterDB called Flushed and Compact and it has a pretty good performance impact. So what is Flushed and Compact? So Flushed and Compact is motivated by a thing that you see in B trees and Bx lawn trees, which is when you do sequential insertions in a B tree, they're more efficient. So let's see why that is. If I start doing some sequential insertions, I start with a key. As we saw before, it goes down the root leaf path and gets put in a leaf. But the thing to notice is that once we've done this first insertion, this root leaf path is in cache. So now when I go to do my next insertion in this sequential workload, it's gonna get to sort of zip right through here without doing AIO. And so all subsequent insertions, all subsequent sequential insertions are gonna be cheaper because they only incur AIO at node boundaries. This also works in Bdebs lawn trees. So here, if I'm doing sequential insertions into a Bdebs lawn tree, I can start doing them. What's gonna happen is they're going to trigger a flush from the root node and that flush is automatically gonna cascade all the way down the tree because all the data is going to the same place. So they'll go right down the tree in a big batch just like you would in a B tree except sort of in a batch rather than individually. That's gonna bring this whole root leaf path into cache as well. So now when I go to do my next set of insertions, they're also gonna be cheaper because they're only gonna incur AIO at node boundaries just like in a B tree. So what happens in a size tiered or a map Bdebs lawn tree? So just like before we want these sort of cheap sequential insertions, see what happens. So if we have this data in a node, say this is like the root node, we've just done a bunch of sequential insertions into the root.  And that sounds great. But suppose that there's already some data that's present in this child, stick a little bit here that's leftover from some earlier work. Any data that's there, we're gonna have to merge again before we can flush it down the tree. And we can still enter performing these compactions or these merges at each level. Maybe some of these things will be in cache, but even if they're in cache, we're gonna be tying up the CPU just going through all this data. And we get this nice big chunk of data that can go right down the tree just adding little bits along the way. So flush and impact tries to address this. So here in this node, we have mostly sequential data that's come in, maybe with a few additional bits in the pieces. And we're gonna flush it down the tree. So this diagram gets really complicated really quickly. So I'm gonna replace all these branch pointers which is the single fat pointer of these branches. It's just supposed to represent all of the pointers. You'll see why it's helpful in a second. So even behind flush to compact is, we first flush references the branches, but we don't immediately compact them. So when we do this, we just add pointers to these branches into the child. And now we're sort of looking at the data through two different windows. So we use metadata in order to make sure that each of the parent and the child sees only the data is supposed to see. So that means that in the parent, the metadata mass means that it only sees the unfushed data and then child, it's only gonna see the flush data. So now if we want to flush from the child to a grandchild, we just go ahead and do that before we do a compaction. So we just again flush these references down as the pointer swing and we use metadata to mask out so that everyone sees what they're supposed to see. Finally, once we've done this, we kick off in queue compactions and all these nodes that have received flush data. So we only initiate this compaction once data sort of reaches its final location.  So if we immediately flush data without doing a compaction, then we save all the work that we were gonna do on compaction. We don't have to do any of that IO. If we had to do IO and if it's in cash, we don't have to do CPU work. We only do it once when it reaches where it's going. And sort of in extreme case of sequential insertions, this helps us get to a right amp of one and also means that we're not using a bunch of CPU resources in order to do this. And that's because that data is just gonna get flushed all the way down the tree and only be compacted in the leaf. There's a side benefit here, which is that we break a serial chain of compactions into a parallel one.  But if we flush before we do all the flushes before we do the compactions, then we can just run them all in parallel. We don't have to have this serialization. And we designed SplinterDB to even allow you to concurrent compactions in each trunk node. So while one compaction is going on a trunk node, you can flush more data into it and start a compaction in that node. And you can have as many as you want in any trunk node. And so both these things should help improve insertion concurrency. All right, so I'm just gonna quickly go through some benchmarks of this because I'm running out of time. So what we do to test the sequential insertion performance is we ran a single-throwed workload where some percentage of the items are sequentially generated and the rest are uniformly random. So this plot shows SplinterDB and RocksDB, the y-axis is the throughput and the x-axis is the percentage of the workload that's sequential. And note this x-axis is very much not to scale. So they'll be fooled into reading some linearity or something into this. And as you can see, SplinterDB smoothly increases its throughput as the workload gets more sequential. That's because of Flushed and Compact. And you also see our improvement from RocksDB but it's not nearly as big as the one for Splinter. I also said that Flushed and Compact improves insertion scaling. And as you can see in this plot here where the x-axis is the number of threads and the y-axis is the throughput, SplinterDB's throughput scales well up until about 12 threads and it's about seven times the throughput of a single thread. At that point you start to level out but that's because SplinterDB is using most of the device bandwidth at that point. All right, so let me come back to this initial slide here. So we kind of walked through this, we spent a lot of time on that, be the epsilon trees and so on. So we can kind of make this into a full cycle now. SplinterDB has been integrated into vSAN 8.0 which just shipped a few weeks ago. So we're really proud of that. We've also, because SplinterDB is a general purpose key value store and not just so specialized solution, we've also open sourced it and you can go download the code and we'd be really thrilled if you, and you guys wanted to either build on SplinterDB or just use it as a comparison point in any work that you guys are doing. We'd love to help if you get stuck or anything like that. And yeah, so I'm just gonna say like going forward, what are some of the sort of lessons that we've learned here? Like we used theory to build, makes learning to be go faster, basically. We used inspiration from theory, and some of my other work also uses theory to build fast hash tables and filters and so on. And I think it's always interesting to ask what other systems could benefit as well, perhaps things with transactions, replication. And I know this is not a theory audience, but like I think to put to the theoreticians out there, like I think the question is, how can we get theory to focus more on problems that directly impact systems? Like what kind of changes to your approach that theory people can make, will make those ideas more applicable? All right, that's the end of my talk. Thank you very much. That's awesome. So I'll follow up half an hour. Rap, show us applause. All right, thank you. All right, thank you so much. It's Zoom, it's gonna, I'm happy. I'm happy too, don't worry. I'm just kidding, I've done a Zoom talk before. Okay, I have maybe one last question from me. Sure. I don't think I was the first to raise my hand so if someone else wants to go first. Everybody else was applauding. Okay, yeah, so I was wondering to what degree is reader-writer concurrency a thing that VSAN cares about? And were there any interesting challenges or solutions that you had to come up with, with the addition of the like the maplets and the, I forgot what they called them, but the additional kinds of buffers. I was basically just wondering like, are there interesting concurrency problems here? Does it scale well to like many readers and many writers and stuff like that? Yeah, that's a great question. So we really cared about that a lot. VSAN cares a huge amount about concurrent readers and writers and we tried to design a system to handle that as possible. It scales really well. One of the nice things about this data structure and I didn't really focus on that is that most of these data structures are immutable. So the branches basically they're immutable until you slice the data out of them. So it's not immutable to deletion but the otherwise just stay the same on disk. And the maplets are also immutable.  So you don't really have a lot of reader-writer conflicts on the core data structure. There's just the, you know, basically the mem table where you're ingesting data before you write it out. That's where you have to handle reader-writer conflicts. And we just have a highly concurrent B tree there that at least so far is working well for our purposes. And we thought about this a lot. We use distributed reader-writer locks and we tried to make sure that our cache is highly concurrent and stuff like this. We spend a lot of time engineering and stuff. So it's a problem that we cared about and we sort of addressed it with those sort of things.


Welcome to Nursing School Explained and another video in the Care Plan Help series where we tie the lab values and the nursing considerations to our patient's specific scenario. This time we'll go over platelets and we know that platelets help us with clotting and stopping any kind of bleeding. So, in case the platelets are off, specifically if they are low, we want to assess the patient's vital signs because when the platelets are low, that means the patient can't clot and they might have some bleeding and that would be reflected in the heart rate and the blood pressure. Certainly we want to check for any signs and symptoms of bleeding from any surgical sites, any invasive sites such as arterial lines or central lines that have been removed, any bleeding of the stool and the gums. When platelets are extremely low, initially it will manifest itself as paticae and then as purpura, which is a pretty late sign, so just make sure that you know what those look like and hopefully somebody, besides doing clinical, will have already gathered that information. And then for medication, so we want to tie in and make sure is the patient taking any NSAIDs, any non-steroidal anti-inflammatory drugs that make them prone for stomach ulcers and possible GI bleeding and are they taking any anti-platelet agents that might also have something to do with the patient's inability to clot. And so now let's try and apply the information here to a specific patient care scenario that you might encounter in clinical. So the example here is a very simple one, a 45-year-old female whose status post-cholestestectomy, her past medical history includes diabetes, hyperlipidemia and she's a smoker. So if we check the vital signs and we see that she's tachycardic and hypotensive and we're worried about her platelets, we can give her IV fluids to help bring the blood pressure up in the heart rate down. We could consider a platelet transfusion if they are low and even a blood transfusion if it has resulted in bleeding and she's now low on her red blood cells and hemoglobin and hematocrit as well. Now if there's any signs and symptoms of bleeding from like a surgical site or from let's say the removal of a central line, we want to perform a dressing change and apply simple pressure in order to help with that. If the patient is on any meds and we notice that there are abnormalities in the platelets, specifically if they are low, we want to definitely hold the medication, assess the lab value first and then notify the provider to maybe consider taking this medication off the patient's list for good because now they're interfering with the platelets. And specifically in this patient with the status postcholestestectomy, so if she had some sort of underlying condition that made her prone to having low platelets, we certainly have to consider that because surgery is going to put the patient at risk for bleeding and if she already goes in with a low platelet level into the surgery, then she's at higher risk of bleeding and some possible complications. So that's why it's so important to look at the patient's specific situation and also their past medical history, their current lab values and tie it all together and think about, critically think about what kind of complications am I looking for because that's our goal. We want to prevent any potential complications in clinical and show our instructor that we're able to relate the theoretical knowledge and the patient that's in front of us and really think about that and prevent any adverse patient outcomes. Thanks for watching this video. Also check out the other videos in the Care Plan Help Series that pertain to lab results and nursing considerations. Thanks for watching.


Welcome back to the Breakfast here on PLOS TV Africa. Earlier, one of the stories that we spoke about in the news, the president, Mohammad Abouari, of course, I was quoted as asking that other nations do better with COVID-19 or support to developing countries. And of course, that includes with vaccines. And of course, that brings us to the conversation on how prepared Nigeria was and is and will be in case there is another pandemic or medical emergency or health emergency in the future. This morning, we're speaking with Dr. Bolaji Obadeyi, who is a member of the governing board, League of State Primary Health Care Agency. Good morning, Dr. Obadeyi. Thanks for joining us. Good morning. It's my pleasure to be here. Great to have you on the program. I'll start with, you know, let's go back to, you know, how we have fared with regards to our response to COVID-19. Do you think or would you say that, you know, League of State and of course Nigeria as a whole has done well enough, you know, with our response to COVID-19 so far? Thank you for that question. I think one can say that relative to the resources that we have, the technology that we have and the manpower, health manpower that we can boast of, I would say that Nigeria and League of State especially have done very well, commendably well with regards to responding to the COVID-19 pandemic. If you recall, last year, 2020, it was completely unexpected and precedent. But somehow, as soon as the news broke in December of 2019 in China, from about January of 2020, League of State through the Commissioner for Health has been mobilizing resources, getting ready, thinking it through, and we're able to put up a good response to the pandemic. If you look at the structure of the pandemic in Nigeria, League of State is the epicenter for this pandemic followed by Abuja and then other hotspots in the country. So if we're able to contain the pandemic in League of State, which we have, then it means that we have done creditably well. We have also received commendation from other international bodies in our ability to prevent the COVID-19 pandemic from spreading across our country and the sub-region. So I would say with the little resources that we have as a developing country, with the manpower constraints that we have also, we've done very well. There's always room for improvement in any enterprise. Yeah, but Dr. Bade, I want you to clarify because I think one thing that we still didn't get to talk about in the period that we've been dealing with COVID-19 is what exactly the reason is that the virus didn't spread as far and didn't become as deadly as other nations had projected for Africa and for Nigeria. Some people would say we probably were lucky. There was even people who said maybe it's the weather here. The weather is too hot for it to survive and some of all of that. But we've never really been able to answer the question as to what made COVID-19 not as deadly as was expected here in Nigeria. Yeah, thank you so much. I think that's a million dollar question. It's a question of many parts. I would say that it's a question that needs to be answered with evidence and science from research. We don't want to postulate and say maybe it's this and maybe it's that. COVID-19 is a disease that's barely two years old. It's going to be two years old this December. So all the research and the knowledge and the evidence base and the data that we have about COVID-19 is all two years, 24 months old. And that's a very short time to begin to make pronouncements about any disease at all, polio, measles and the others go back up to about a hundred years. So this is something a disease I think we need to build data and evidence about and the research is ongoing. There is a research plan within the legal state response system that looks into the evidence and data that is evolving locally from the COVID-19 response. And I'm sure that all over the country in other states as well as the federal level is a lot of research and thinking going on. I would say though that the public health measures we put in place across Nigeria were quick. They were decisive. And I think that's a small part of why the COVID-19 pandemic did not spread as much as it did. Nonetheless, there's still a lot of research just as you say to look into genetics, to look into the effect of the weather, to look into the effect of cross immunity from other diseases that are endemic around us. What for example is the role of our immunity, long-standing immunity to malaria in protecting us against COVID-19. These are all hypotheses. So research I think in the coming years will confirm definitively what are the reasons why the pandemic has not spread. Okay, so let's go back in time now. Looking at the time when we had the Ebola outbreak now we have COVID-19. Do you think that we have done better? Because some people have the opinion that we practically managed the Ebola outbreak than the COVID-19. What are your thoughts? I think those are two different templates. The Ebola virus and the pathophysiology of that disease is different from that of COVID-19. But where we have similarities I will tell you. I think the similarities that we have with Ebola and COVID-19 is that in both instances we had extremely strong leadership and governance. In the case of Ebola when that happened the government in the first case was in legal state. The government immediately kicked into action. The governor for the state came back from a trip. The commissioner on the ground was all over the place doing what needed to be done. And again with COVID-19 I think leadership has been strong in this particular health crisis with the commissioner and with the governor taking charge immediately and working there tonight to ensure that things went well. So while those two diseases are not exactly comparable, the case fatality for Ebola is much higher than the one for COVID-19. The means of transmission is different. The approach to prevention and control is also slightly different. So while there are different diseases you know with any emergency at all or any outbreak or any biosecurity threats leadership is extremely critical. Governance is extremely critical and in both instances I think we have been very lucky with leadership and governance. But another issue that is very strong is the issue of acceptance and that has to do with awareness because if you remember the time of the outbreak of Ebola we found out that the awareness was very strong compared to that of COVID-19 and then we had different conspiracy theory of how to stop the Ebola issues. Now let's go back to what you mentioned. You talked about the Nigerian government acting in December. I'm thinking that we started taking action in March. That's when we started seeing deliberate steps in action. So yes it broke in December you know in China there were about January we were a little bit not too you know strong with our policies and action February up until March and that's when we started having the shutdown and lockdown across spot. So what is do you think that in terms of awareness we're creating enough awareness you know with the pandemic's you know situation and in the nearest future what are we going to do to create awareness? Excellent you know the key word there is visibility there is a lot of action good technical action going on in the background that the public might not be privy to in terms of what protocols can we develop what handbooks do we have? Where do we mobilize resources from? What are the what-if scenarios? What if this happens? How do we respond to that? What if that happens? How do we respond to that? Constituting the several pillars of the COVID-19 response those are things that were going on the background that the public was not involved in for at least two months before at least a clear month before the first case we had in Lagos state with COVID-19. In looking at communication for the two diseases comparing Ebola and COVID-19 you know with Ebola it's a more dramatic disease that that you don't need to tell anybody do you really want to get this because there's bleeding from the earifices there's fever it's toxic the person dies rapidly with COVID-19 we even had so many asymptomatic cases if you recall and that means that for the average Nigerian or the market woman when you say asymptomatic the understanding is not there the person's understanding of asymptomatic is zero infection whereas in public health we know that asymptomatic means you carry the virus you are able to infect all that people even though you are not showing any symptoms so that's that's where risk communication we need to improve our skills as medical professionals as sociologists as psychologists how we communicate the risk of these various different diseases to people so they have understanding that while it's not like Ebola where there's bleeding and there's death and you're very ill COVID-19 if you are not serious if you get infected it can kill you as well so that's an important part in terms of preparedness we often talk about the three aims we talk about money we talk about materials and we talk about mom power those three are pre-kable for Nigeria in how we prepare for future biosecurity threats these are biological emergencies this is that I'll call as a result of changes in our ecosystem in the climate change the rising temperature the fact that there's urbanization which means we're now in close proximity to animals that we never used to relate to it so I think there will be future pandemics there will definitely be more epidemics in Nigeria we're dealing with about five different epidemics as we speak it's not just COVID-19 there's LASA fever there is a yellow fever there are breaks of cholera there's cow pox quite quite a basket of things that we're having to respond to and so I think we need to have a strategy as a country in terms of how we want to this is the new normal and it's not it's not a very comfortable normal how do we move forward as Nigeria so money manpower materials that's conventional the non-conventional tool to add will be technology digital technology vaccine technology and of course lastly leadership and governance we often forget that I say governments because when you look at how we were able to mobilize against COVID-19 um we can have the ministry at the Secretariat can have the federal government in Abuja we still need other traditional lower government structures to reach our people the market management governance the local government health authorities the local government chairman will leverage all of these platforms platforms in communicating to people about what to do against COVID-19 so I think in preparing for future pandemics the three conventional ones morning we can't avoid that materials we need to invest in our health system manpower we need to develop new streams of healthcare workers more public health practitioners more molecular biologists more epidemiologists more people in the area of testing which we have to rapidly expand capacity and so on we also need to invest in technology this technology e-health invest in vaccine technology possibly producing our own vaccines ourselves here in Nigeria and of course we need to strengthen leadership and governance that I think is the way to good fabric with this unseen enemy oh dr buddy it's um well I'm not sure when we will get to that stage where we start to you know prepare our own vaccines here in Nigeria might take a while you know seeing our investments in healthcare from the national level and on the state level also but I want you to break down the you know the discussion and preparedness you know in layman terms if possible um to understand better what needs to be done because um the thing with pandemics and health crisis is you can never really tell if it's going to be one that attacks blood cells or one that attacks you know the respiratory system or one that attacks the gastrointestinal tract you can never really tell um and and so they have to be systems that are put in place that would be you know the quick response you know that you know that the the the state would you know would put in into action um in case there is a pandemic like that so what have you seen that we still lack here in Lagos and across Nigeria that you believe must be immediately um properly invested in um to prepare ourselves for another pandemic if if if need be okay um that's a very broad question but I will come at I will approach it from the angle of what I do I'm a health economist so I will focus for example on financing health financing by the way we used to produce vaccines in Nigeria in the past and I think we can do it again if we just get dedicated government support the UK government as soon as the COVID-19 pandemic broke provided I think 10 billion pounds dedicated to just developing the vaccine so that's the kind of approach we need to have before cost and um and put our money where our mouth is okay on visual financing just that alone there's government financing of health which needs to improve that is the amount of money that government invests in health not just that government also needs to look at um a performance-based financing what are we getting for the money that we're putting in health that is are we putting money in the right places even more important and there to my heart is the issue of health insurance that is now household financing or personal financing okay if we say government has done his best government cannot do more there's no more money government can spend more can individuals spend more can we spend that money in a strategic way can we rather than pay out of pocket which means in the pandemic you have your money under your pillow and you're waiting for when you're in and then you just put together what you have and go to the hospital which is unlikely to be love because nobody keeps money for sickness so in terms of the layman that is what can I do I would say individual Nigerians should begin to look more at EPA schemes at having a basic health insurance package that is subscribing to whatever you can afford in the various options which are available to us now whether government health insurance or private health insurance subscribing to one to such an extent that if there's an outbreak of a pandemic or an epidemic and it affects your family you can access health care you simply go there with your electronic card you don't need to look for money at that time so that's where we an individual or a family can prepare for a pandemic having some kind of cover that gives them access to health care services not on the basis of ability to pay but on the basis of their need so when I pick up my card I go to the hospital and there are various prepaid schemes which are available now that are quite affordable so in terms of financing that's one thing we can do breaking it breaking it down again in terms of human resources and manpower we need to take care of what we have frankly I don't know how we can do that to prevent our doctors and nurses from going abroad but not everybody's living because of money some people are living because they're not fulfilled because they can't practice what they've learned so if we invest more in the health system our doctors and nurses perhaps those who just want to who have an altruistic motive will stay and and do their bills we also need to train more a country of 220 million people uh one of our former professors professor what they talk about Lucas god bless his soul used to say that our problem is not brain drain is that we're not training enough nurses and doctors he thinks that as a matter of fact we are able to export even if we trained enough like the Philippines like Cuba and other India and other countries are doing so we need to train more and more help manpower of the right cater what everybody needs to be a doctor you can be a community health worker if that's a level of education that you have you can be a nurse you can be a pharmacist there's so much of the real skaters that we need so I think those are simple areas where the the regular man can be the average Nigerian can begin to think about how to prepare my family for another different outbreak of disease which I think will happen everybody should have some form of cover some form of insurance cover yeah it's a lot it's a lot of work obviously it's um it will take a while before we get there all right but let's also uh look at the basic structures you know I know that you've mentioned that money is very important and of course money would answer to some of you know the things that we will need including the structures themselves but let's assume that we have the money what should we be channeling the money to I mean we're talking about structures now for instance uh when we had the COVID-19 outbreak we started battling and grappling with the issue of isolation centers in a lot of states that really there was no provision for isolation centers and then state governors started you know seeing how to pump monies and you know create isolation centers so my question now is what basic infrastructures do we need as a country across the 36 states including the FCT you know in case there's a pandemic there's an emergency what basic infrastructures do we need across the entire country thank you so much that's a question we need to ask ourselves from time to time there are three tiers of the health system the base the foundation of the health system is the primary health care system that's the entry level that's a touch point for more than 70 percent of Nigerians that's the one that is closest to most Nigerians is the place where you can get the the fastest health care possible above the primary health care level is the secondary level whatever can be handled at the primary health care level goes to secondary level what can be handled at secondary level goes to tertiary level I think if that's the way that the pyramid is built the investment into the health sector should go to the base you don't build the house and put all the money on the roof we build the house and ensure that the foundation is strong enough and that's the same approach we should have to strengthen our health care system investing in the place where 70 percent of Nigerians access their health care from making sure that the primary health care system works making sure that the primary health care system has the approach that is preventive the public health approach clinical medicine attends to one person at a time if you go to see your specialist your gynecologist or your new surgeon there can be ten of you in the room is one person at a time public health is population health it's the branch of medicine that looks at the health of groups of people in terms of preventing diseases and controlling the outbreak of disease that's another area that we need to invest our money in ensuring that we have a strong public health system that our epidemiological skills and knowledge is strengthened so that when we have these outbreaks which we have all the time cholera here and there yellow fever outbreak here and there we have the right manpower to respond to those outbreaks what we see unfortunately is that we invest in the roof we invest in the tertiary system we should don't get me wrong we should have inacular rituals we should have MRI machines we should have CT scans but if you decorate the roof and the foundation is crumbling what we've seen is what we're going to continue to see so I think we need to look at the allocative efficiency of our health system where should we put more money it's the base it's the foundation and showing that there is a health post health center that is functional well-staffed well-equipped has drugs that is close to everybody so that if an epidemic breaks we are not then as you say asking ourselves where do we put these people they simply go to their nearest functioning health center when we have a designated isolation bed at least one or two in each of those places where we can put people who have a bit kind of condition but if we don't have that then what we're going to see is the last village effort to quickly set up isolation centers having said that let's be credit to all the state governments every single one of them COVID-19 has brought out some of the best in us I think that every state has cannot boast of at least one if not two or three designated isolation centers that those things were not there two years ago so I think we're moving in the right direction we simply need to institutionalize the COVID-19 response to ensure that the capacity is spread broadly across the health system using of course the primary health care system as the beast Dr. Balaji Obadeyi a member of the governing board lego state primary health care agency very very interesting conversation and I will say thank you for your time this morning thank you for chatting with us wish you a beautiful weekend ahead thank you very much thank you for having me it's a pleasure and that's where we'll be saying goodbye this Friday morning thank you very much for being with us all the way since seven o'clock in the morning uh if you missed out on any parts of our conversation remember where to catch up it's simply at plus TV Africa of course on facebook and instagram and of youtube as well plus TV Africa and we wish you a very great weekend I am messy and I am a


I've been a sheep shearer all my life for the 20-odd years of it and then I got to the age where I trained disengaged young Indigenous kids the art of shearing and it was then that I learnt that I liked to help. I applied for a police liaison job at Kalamala and I got it and I've done that for four years, loved it and they encouraged me to apply to be a sworn officer. The more Indigenous people we've got in the QPS or in those types of jobs it's going to help our communities, our youth will see realistic role models walking around country communities. I believe we'll see less crime in those Indigenous areas where maybe the kids may feel comfortable to come up and have a yarn. I know they do to me, I love it, recommend it to anybody and I do to anybody that I come across.


I went to school and got into high level and talked to the little ones, well all the different classes, how we went, what kind of like the school that we went to when we started school and it was nice to talk to the kids to tell them that how I started school and how important the education is nowadays you know because them days when I started school I couldn't speak a word of English so I went to school for four years and then I left school. My language was Cree yeah it's it's Cree and then I that's all the language we knew I couldn't speak a word of English I didn't know yes or no and I started school and I came to move to we were out of town and we moved into town and then we had to start school because we were getting family allowance then we went to school and we go home for lunch if we had something to eat there that is and uh and then we go back yes we went to school I couldn't understand the teacher nor the teachers couldn't understand me because I was speaking Cree and and uh it was a bad experience I never liked that part but I mean it took a little while before I was able to say a few words in English and then from there you learn from the kids other kids teach you you know like you know so that's I went from there and that's why I have hard times speaking English now because uh I still think that when I speak English people laugh at me you know so I'm always a little bit shy to uh to speak English well actually the little kids didn't ask too much too many questions they just asked me what grade I went to I think and what kind of school I went to and what I had for a teacher and what we did in school and that was it mostly I just told them that I had a bad experience because I couldn't speak English and that I had to and I was already coming nine years old so it was but I picked up English and and that's and and another thing that the kids uh they asked me I play a little bit of music I play fiddle so I play fiddle for these kids every class I go to them and they ask me questions and then ask me how how long I've been playing violin or how old when I was starting and so I play a fiddle for them and they enjoy that they even dance so I think I think mostly to talk about education is so important nowadays it's very very important because us us people nowadays if we have to make living off the land we can do it we cannot make it because we have to have we have to be educated to be able to get good jobs and that's only that's only life nowadays otherwise it's it's it's nothing for for the people like they have to they have to go to school and they have to learn and and then they can get a job and once once you get the education you got it made you know because then you can find a job then you know you can keep on going to school and finally you can you know uh tell the what you are going to say was uh you know once you get that education and keep going to school and finally know what you want to do with your life then you go from there it would probably help them because they kind of if they understand what I was trying to say to them they probably would would have an idea what what could be done and once you can you can do anything in this life if you have a chance to do it you can do anything you want to be with education especially so if the kids want to like they love music if they want to play fiddle it's there they can learn it now it's and my myself with we just had no entertainment no new tv them days so we just learned to play music play fiddle and I think it helped the kids no they had they had a place set up in the gym and but first we went there they set up lunch for us they gave us lunch and then they tell me they tell us to go what class we'll come to our in the gym what class is going to be there first then we start from the little ones and then go up to high rates I think I had about three four classes there you know like come take big big turns coming and then in the end they all get together and they they dance around there so yeah it is because of the music that you yeah the music I gave them they enjoyed that very much and they asked me how long have you been playing when do you start playing I should I start playing I was eight years old I said and so I play whatever they want to hear so you were self-taught yes by year yeah I think you're they're willing to listen to what we have to say they want to keep listening they they really thank me very much for coming out and be able to talk to the kids a little bit I didn't talk as much as my wife did but see I I didn't mostly they're fiddling in tell them what education is all about I feel quite comfortable because I'm telling the truth when you're telling the truth you know it's easy to speak out well I I just think I just I just think we hit that part a little bit about education was important like we can't keep on living the way we used to live off the land and we can't do that I think it's mostly we have to go the way the white man teaching us to do it now but we can stick still stay with our culture and stuff and hunt or if we get chance but now we're not even hunting hardly because they we don't have a place to hunt because the land is all taken up pretty much in our area anyway to me I can't see it because I don't know what good it's gonna do anybody because we're not gonna ever use it even if we go through it I don't see how we could use that and what's used to train people in our way when we're we have to live in the different different style of life like like we can't hunt we can't trap and that's our life to trap and hadn't move around that was our life but now we can't do that we have to we have to stay one place and stay stay there and and and live like the way the other people live keep going keep going to school and make yourself some something out of yourself that's the only way now otherwise we got other people coming into our land and and taking over and we're we're left with nothing we cannot how are we gonna live here education the most important thing in our in our time year now and then to go forward with it it's I can't see any other any other thing any other way it's easy now for kids because the kids already have lots of experience they can speak English from the time they're born and I had eight children and I did not want them to speak Cree because people we couldn't speak we you know had such tough time trying to speak Cree with you know I mean English not Cree but from Cree to English to change that like and actually English is very easy to learn but that's why I never taught my kids to speak Cree and I'm kind of sorry now because they should they should be able to speak Cree that language I think is all right if because people can still talk is the language you know I mean that's wrong I don't see anything wrong with that I I started out the hard way I started working I never trapped in my life and I liked hunt but I never trapped I never because I know trapping was not good and I see there was not going to be trapped at all times because there's so many people how where is the animals going to come from to kill off the animals and I didn't like it I didn't want to do it so I started working and I started working when I was coming 15 years oldness in the Swami login camps and I worked till I was let's see 1979 I think I was 59 years old or 49 years old something like that and anyway then that's only that's only way you gotta that's only thing I can say that that's how I started out and that's the only way you can do it no other way I can't see it me other way I would be good if they could see what I if I could tell my whole life story would but I haven't got time for that so it's a long long road like I want to learn to I know how to make moose hides but I never really entered but I helped making moose hides and make dry meat and I can do that yet I would like to pass that on you know I you know you I don't know how to cut the meat you're gonna make dry meat you buy dry meat in the storage it's not cut right you can't eat it it's like soon rubber I would like to see more our people getting into the law and and and become leaders in the governments and get into that stuff so we can all work together I think it would be better for our people yeah instead of getting other people from different countries and trying to bring them here and start to run our country it's a they say this is our oh canada we've had sense to play this for the for the uh for the students but I did play one called lonesome lonesome indian okay awesome you want to play a little bit of that