Hi, my name is Bartosz. I'm with Baidibre French-based Linux consultancy and today I'll be talking about why so many Linux plumbing layer components end up having multiple incompatible major versions and what can we do to avoid it. I'll be mostly talking about very specific things that make kernel or user space library interfaces impossible to improve without breaking compatibility contracts in general what mistakes one should try to avoid in order to keep libraries APIs in general extendable in the future. Baidibre is not so small now. We are around 50 people. We are a Linux consultancy based in southern France. We help clients launch hardware and software products and we help them through the entire process from concept to manufacturing. We work with the Linux community extensively. We upstream support for several hardware platforms and we are the founding members of the kernel CI project. Personally I have 12 years of experience now. I work all across the stack from kernel space and even Bermedal and bootloader's RTOS up to low-level user space and Linux build tools like Buildroot or Yocto. I am the author and maintainer of LibGPID and co-maintener of the GPIO kernel subsystem and I also contribute a lot to different open source projects including Yocto. This talk is based on lessons learned while working on two open source components. The GPIO character device exposed by Linux for every GPIO chip since 4.6 and the user space library and tools that we provide in order to wrap the kernel interfaces in an elegant and easy to use way. A bit of history first. The GPIO character device has been introduced in the kernel as a replacement for the deprecated CISFS interface and LibGPID was released in order to provide a C interface that is easier to use than role in a PsiOctol calls and that would be the base for the command line tools that could be used in scripts with the idea that this would make it easier for people used to CISFS to switch to character device. And if you look at the UAPI header for the GPIO character device today you'll see that the number of symbols is marked as GPIO underscore V2 and if you look at the mainline git repository for LibGPID at kernel.org you'll notice that there is a branch called next slash LibGPID V2 and that it is under active development at the moment. So what exactly happened? Why are we creating a new backward incompatible interface for GPIO only four years after the first one was released? So before I tell you I'd like to discuss what different types of compatibility we are exposed to in Linux. Some of those things I described may be obvious to many folks but there are unexpected caveats when it comes to binary compatibility that even experienced Linux hackers find surprising especially if you venture beyond C and go into the realm of C++ or Rust. And there are also a lot of kernel developers who rarely venture into the user land where issues of compatibility are more pronounced than what you deal with in the kernel. In the kernel it's quite simple, there's no stability of internal interfaces and indefinite stability of kernel user space interfaces but this is actually a bit more complicated than that in the user space. So let's take LibGPID as an example. LibGPID talks to the kernel using Ioctl system calls passing around a set of structures defined in the relevant user API, in the relevant user API of the exposed by kernel and on the other side it exposes a C header containing a bunch of symbols and a shared library called LibGPID. On top of that we've built the C++ and Python bindings and well Python bindings are written in C and C++ bindings are written in C++. So the important thing to note is that the C++ bindings expose another shared library called LibGPID CXX to make things complicated. But this is all pretty normal stuff, most system-level components are written in C and compiled as shared libraries and expose C headers to interface with the library and while the C standard does not define any ABI standard Linux has a de facto standardized ABI. So we're now concerned with two types of compatibility, source or API and binary or ABI. The general definitions of the two are simple. For the former two versions of a library are source compatible if any program using the exposed symbols can be recompiled with both without updating its source. And for the latter it's a bit stricter, a program linked against the library is guaranteed to work with both versions without rebuilding the program for the versions to be ABI compatible. And so it truly is like both ABI and ABI truly are just contracts and this contract can be honored, extended or broken between versions. And when considering the differences between two versions we can have three situations both for API as well as ABI compatibility. So we can go from A to B or one to two without any change in the interface, for example the only thing that changed is some internal implementation details. And we can go from A to B and add new interfaces, for example we add a new function that makes A compatible with B, so program using A will work with B but programs using B are not guaranteed to work with A. And finally we can go from A to B and remove a certain function or replace it with a different one or change the arguments or modify a visible structure or change in values and in that case programs, in that case we break all compatibility and programs using A will not work with B and vice versa. So API and ABI compatibility are independent from each other and one can be broken without affecting the other. For instance if you extend the size of a struct but don't change its previous layout or rename the numbers, you do break the ABI but you will still be source compatible. If you on the other hand rename some preprocessor defines in the header but keep the old values or rename enums but keep the old values, you do break the source compatibility but not the ABI because a program linked against two versions with two versions of libraries built with different symbol names that retain the values will work. And so back to live GPIOD and the character device. When the first version of the character device and subsequently the first version of live GPIOD was released it supported a set of features that we wanted to expose to the user space because obviously we knew what should be available to user land and with GPIO features should remain confined to the kernel except that those SQ users started coming up with annoyingly valid reasons for extending the interface with features that we didn't plan for and then we faced the reality of our interfaces both in the kernel as well as in live GPIOD being badly designed on several levels. So example in the structure passed between the kernel and the user space we left no reserved space for future use while we could potentially add new flags like new values for the flags it wouldn't be enough because users at some point started asking us to add for instance the debounce period setting because what is reading line interrupts in user space good for if you can't set up the debouncing. Another request for line events was sequence numbers because what good is bitbanging implementation in user space if you need that if you can't assure the right ordering of events in the queue. So already we are in a situation where we could introduce new feature we could not introduce new features to the interface that would not break the ABI contract but we were in the situation where we could introduce new features to the interface but that would but we basically locked ourselves out of that possibility because we didn't think about it in advance and we didn't simply leave any reserved space. So this is the lesson one that we learned always add reserved space to your exposed structures. It costs very little unless you go above and know that the memory page boundary it unless you do that it doesn't really cost anything and it allows you to extend your interfaces in the future without breaking the contract especially when designing eye octals because in user space we could we can do a better thing that I will discuss in a minute but lesson one learned in lib GPOD when sorry in the GPIO character device interface when we designed it we left some padding so that well right now it seems as if we are done that the API is complete we really don't want to expose any new features but I have a feeling that someone at some point will come up with an idea that we will agree to implement and for that we're already prepared every structure that we are exposing in the V2 interface has already some padding we were we should be good for most part. So lesson two is related to to the problem described before as you could see we have so if you look into the header for lib GPOD you're gonna notice that we have the same problem several structures are exposed as in their layout is defined in the header that means that the user can allocate the structure on the stack and their layout is exposed they are exposed because at the time this was my first serious library that I was designing I didn't know any better and I feared that making certain structures opaque would impose a penalty if they were allocated on the heap exclusively like for like for instance the mentioned line event structures they normally would need to be created a lot because there can be a lot of line events coming up in the queue and other concern was that they that if I made certain structures opaque they would be very complicated to use with C accessors and in this case I'm talking mostly about the config structure that is used to set up a line request a GPL and line request before translating it into the kernel version and passing it over by over the I have to call so I settled on having certain structures visible to the user and of course this was a bad idea in fact it's a well-known pattern in low level Linux libraries to make all objects opaque and this really is the right approach as not making any guarantees to the users about the layout of or the size of structures saves us all the all kinds of problems with the ABI but what if you know that certain structures will be allocated a lot and you don't want to impact the performance by calling into the allocator all the time do one of the ideas and this is what we used in libGPairD is using opaque containers so we have a container object that is allocated on the heap and then when we and then it stores static like it stores already pre-allocated structures that we can reuse all the time and they are not being reallocated again the lesson learned so this is maybe an example that gives you an idea of what we did so before above this is what we had we had a user who wanted to create a buffer for line events needed to allocate it on the crude allocated on the on the stack and then simply call a function that would read the events and try to translate it then from the kernel form into the user space for and put them in store them into into this visible buffer so what we do now is that the user now has to allocate a an edge event buffer and then read events into that buffer and then this is the only time that we actually allocate anything here and then when the events are are read they are stored in this buffer and the user can actually get handles to the opaque event structures and act on them without reallocating them all the time so this is obviously an improvement but what about other languages so we also support Python C++ for Python it's clear Python doesn't have API and also in Python every object is allocated on the heap so this this is not really a concern here but in C++ the class layout is visible and in this case we just went with the pimple idiom so that classes look like this they expose only their methods and use a smart pointer to store a pointer to the implementation class which is all which is actually hidden so yeah the the the the the class the public part of the class only contains a smart pointer and the implementation itself is hidden from from the from the user and what we do for edge events in this case because we wanted to replicate this very efficient way of allocating the event buffer we actually used a bit of polymorphism where the the edge event classes that are stored in the buffer are not managed but as soon as the user copies them as long as the user just reads their values that that's fine they're not being copied but as soon as the user copies them then they become managed with with a smart pointer so just a nice trick to have something efficient replicated sufficient behavior in C++ so lessons 3 make sure your expose structures work on 32 and 64-bit architectures so this I'm adding the asterisk here because while for most part you don't need to be concerned about the layout of structures as long as you're not sending them over the wire when you're defining them but even then you if you're sending them over the wire you should probably not really do it yourself you should use some standard way of marshalling the data but in general when you're defining a structure you're not really concerned with padding with alignment and all that this is done by the compiler except when okay except yeah except in certain surprising circumstances so let's take a look at this structure so what is wrong with that so it only has two fields and this is the structure exposed by the version one of the criminal interface for the GPIO character device this is the event data that would be actually read by the GPIO D from the kernel so what is wrong with it at first sight nothing is wrong with it on 32-bit architectures the structure will be aligned to 32 bits and this if we look just like this and on 64-bit architectures there would be additional padding at it so that the structure would be aligned to 64 bits and this isn't a problem if your kernel is 64 bits and the user space is 64 bits but we also need to support Linux that support compatibility mode for 32-bit user space that the kernel is 64 bits and the user space is compiled in as 32 bits so we actually encountered this situation where we would send events from 64-bit kernel to 32-bit user space and the user space would be completely confused because the kernel would send structures that would be 128 bits in size and it wouldn't expect it it would expect 96 bits so how do you fix that all again just make sure that you pad your structure to the correct size especially if this is the structure that is exposed by the kernel you have to think about the compatibility mode for v2 we made sure that all our structures especially those that can be retrieved with the read system call be correctly padded and work with this specific compatibility case next lesson some surprising details can break the ABI so it turns out that the structure layout is not the only thing that's considered to be part of the contract between the kernel and the user space so everyone knows one of the most famous Linux runs about how we must not suddenly change the error codes returned to user space in hindsight this is a no-brainer however we had a quite a similar situation over at the GPIO subsystem so the clock that we used to timestamp the line events in the kernel that would be then exported to user space was the real-time clock at some point it was determined that it uh oh it's always that a real clock is not a real-time clock is not is not guaranteed to not go back or to move back and forth in time and that it wouldn't really so users notice that we don't really convey the right information to the user space about the timestamps of the events or approximate moments in time when the events would occur so we decided to change it to the monotonic clock and soon after we had a user complaint about the change because they actually relied on the timestamps being derived from the real-time clock fortunately the user adjusted their code and didn't require us to go back but if they insisted obviously we would need to keep the interface stable and you would have to revert that change so fortunately we are still uh we still have uh we retained the monotonic clock but what we did for version two we simply added a switch so that the user can can decipher themselves what clock they want to use so right now we support the monotonic clock uh for event timestamps real-time clock and also we have an upcoming support for for the hardware timestamping engine in it's it's a new general thing in feature in the Linux kernel but we will support it in GPIO for our event timestamps so lesson five don't try to keep the user space data models the same as the one in the kernel so the goal of user space libraries working close to the kernel is to represent whatever the abstract data model kernel uses in a way best suited to user line and oftentimes the kernel representation is quite specific to the problems faced inside the kernel the problems may be different in user space and also user space accesses the data structures in a different way so as tempting as it may be to stick to the to a similar representation it's often better to spend some time on designing a bespoke model of the data structures for for the user space so what you see here is the representation of the GPIOs in the kernel so every chip like we have an example chip that has exports eight lines and what drivers do in the kernel they they call GPIOD get and they receive a GPIO descriptor which is a pointer to a structure defined in the lib GPIOD code and also users can can call a variant of this function called GPIOD get array and receive a container for multiple descriptors what happens now is that the GPIOD get function actually does request the line so that the lines become owned their ownership is passed to the using driver and then nobody else can can request them so this is this is what we do in the kernel and what I thought would be best for version one of lib GPIOD would be to simply follow that model except that there is a slight change so we have we still have a chip and this is the chip that we open with GPIOD chip open unlike the kernel in the kernel the users don't really need to open a chip but in this case we are opening a character device so we have another layer and then a user can call a function called GPIOD chip get line to retrieve an opaque pointer to to the line object and every line object represents a line and now this line is not yet requested this is unlike what the kernel does this line is not yet requested right now we only have a handle that allows us to read information about the line so so we can use we can read its state its direction its various flags and we also can retrieve multiple lines and then then they are stored in an object that we call the GPIOD line bulk and the same they are still not requested so now only now the user can call the request function GPIOD line request or request lines and only then they become owned by the user space process so what is wrong with this so the thing about the line information that can be retrieved from the GPIOD line object is that it's when the when the user space reads it from the kernel it's it really is only a snapshot so we when you read a single flag of the line a single setting of the line you're it's not guaranteed to retain it this value you know in in any time in the future so this is a problem because there is so until why why we do have a certain way of notifying the user space about changes in status of lines it's not very obvious from the API of the library and you would have to unless you set up an event loop if you just have a very simple program you would have to reread the status of the lines manually so we decided for version two to have a completely different model of of the like a completely different data structure model but we still have a chip we still open it with GPIOD chip open but what happens now there's there no longer exists a a a class of objects that would represent lines what we have instead is we have objects called GPIOD line info and it's very explicit that they don't contain up-to-date info that they contain a snapshot of information retrieved at the time when the call was made to GPIOD chip get line info so this is how this is this is an improvement over over the previous version because right now it's very clear that all the settings that the user can read all the flags that or various values that the user can read from this line info object are only a snapshot in time and every time so we still have the mechanism to inform users about changes in the status but right now they receive every time a status occurs they can read a new line infrastructure with up-to-date information so yeah this this is how it works right now for the line info and also the request line requests are made in a different way so we are no longer operating on separate lines because it no longer makes any sense we request entire sets of lines we have a function called GPIOD chip request lines it takes as argument it takes a like an elaborate config structure but right now we have we like we do the request for a set of offsets it can be a single line but we're never operating on single lines we're always doing request for multiple lines at once and this allows us to so now the lifetime of the GPIOD line request object is detached from from the chip so the chip can be closed and the request can be kept because it's actually associated with a different file descriptor so this is this is an idea on how we completely detached our user space data model from the one in the kernel so i i encourage everyone to just try to give it a try not to follow the kernel data structure model to the letter and so the things i described here are very specific to our project i i hope they're they're they're useful but they're pretty specific to this very very narrow set of interfaces but there is this project that i actually i i i sadly learned about it after i designed and released the the first version of the GPIOD but it actually i'm talking about the libabc project which is a a dummy library that simply shows the good practices of designing user space libraries for the plumbing layer in Linux so it contains some things about which i which i spoke so that the that the all data structures exposed by the library should be opaque and should be accessed with dedicated functions that there should be no global state in a library because this way it can be made not not thread safe but thread aware etc etc so this is a good read i i posted the links to it here and that's it for the presentation i'm open to discussion thank you very much i hope this was useful the things i i spoke about here are actually something that i learned the hard way um and i really hope that the second version of libgpav once we release it will be at least resistant to needing to have another incompatible version in the future thank you


Hello, my name is Joseph Bassack. I'm a software engineer at Facebook and a longtime kernel developer and maintainer of ButterFS. Today, I'm going to talk to you a little bit about deploying ButterFS at Facebook scale and some of the lessons we learned and the variety of ways that we use ButterFS in our infrastructure. So first, I can go a little bit of background on Facebook's infrastructure. There's quite a bit of different use cases. As you can imagine, Facebook is a very large company with a lot of different moving parts. So in order to help understand the presentation better, I'm going to kind of walk you through some of the basics of how we have things set up, some of the terminology that we use inside of our infrastructure. And so just to help make it easier to understand the different terminology that I use throughout the presentation. Then I'll go over some of the big use cases that we have for ButterFS. Obviously, every file system has its own pros and cons. This is not to say that ButterFS is perfect for all use cases, but there have been many use cases that we found it to be very valuable. And so I'll kind of describe those in detail and the things that we ran into and just some interesting tidbits and places where ButterFS really helped us versus places where it wasn't so great. The other thing that I will talk about is kind of the things that we did. Interesting things that we learned along the way, things that went really, really well. And also things that didn't go so well and some of the future work and just the direction that we're going and the things that need to be dealt with. So the first is just some background in Facebook infrastructure. Generally speaking, every service that runs inside of Facebook is contained within a container. This is to make it easy for us to migrate. So we have, obviously, we have a lot of machines, more than most. Because of this, it's difficult to manage. We have so many machines, it's impossible to manage in any sort of unique way. So we try to make sure that our machines are relatively consistent across the entire fleet. And thus, the services need to be able to move to any machine at any given point. We do things often like bring down entire data centers to validate our disaster recovery modes and that sort of thing. And so part of what allows us to do that is being able to containerize and ship services around anywhere within the fleet. When I originally joined Facebook in 2014, 2013, a long time ago, the every service had its own dedicated set of machines. So the web services all ran on web service machines. And if you wanted more machines, you had to go to the capacity people and say, listen, I need X thousand more machines or X hundred thousand more machines or whatever because of peak loads. And that's the other thing that's important in Facebook. Not everybody's on Facebook all the time. We have peak times. And so we have capacity set aside to handle peak traffic. But generally speaking, we're relatively unloaded. So anyway, before we had dedicated service, dedicated pools, and so there weren't really containers. It was just the web machines got their web, new versions of HHVM just pushed directly to them. HHVM would restart and they'd keep on going. Same goes for database, memcache, that sort of thing. As we have invested more and more time and energy into resource isolation and containerization, this has changed. So now our largest pool of machines is called TWShared, our container service is called Tupperware. And so the largest chunk of machines that we have is TWShared. And basically every service that doesn't require dedicated machinery exists in this pool of machines. So the web services, messenger services, that sort of thing, before web was on its own set of hardware, now it just exists like any other service inside the set of TWShared. Machines, there are some services that don't do this still like the database, things like MySQL, we don't containerize that. It's, they have their own machines with a special hardware, they get upgraded like any other kind of like small business sort of service that you would expect, upgrades, you know, restarting that sort of stuff. Not a lot of mobility, but that represents a very, very, very small percentage of the fleet nowadays. Nowadays everything is container. And even if you still have your own set of dedicated hardware, like our build servers, it's still built on this Tupperware platform where the service is containerized and ships around. This has obviously gone through a certain amount of evolution over the years. The original version that this looked like was a ButterVest loopback devices. So we had a base image of CentOS and then this base image would be generated with the service specific information and code. And then we would compress this all up and ship it around. And so any, so any box running a service would often have a three or four or five ButterVest file systems running on loopback device within it as part of the container. As we've made improvements, as we've done thing, this existed because as I said, we have a lot of machines and reformatting all of those machines is kind of a big task. And they were all running with them. EXT4 is the root file system. And so early on to validate using ButterVest in this way, it was simpler to just ship loopback devices as ButterVest around onto EXT4 root file systems. And then as we converted more and more machines to ButterVest root devices, we were able to take advantage of more of the interesting features of ButterVest. And just as a quick rundown of ButterVest, in case you haven't paying attention to Linux file systems recently or for the last 10 years, ButterVest is kind of an iteration on Linux file systems. We have built-in snapshotting and compression and individual volume management. So those are the sort of features that were very attractive to the various teams building out the container infrastructure inside Facebook. So one of the very first use cases where ButterVest was sought after for one of its core features was our build servers. Obviously we have a lot of code inside Facebook. This is not only like the webpage that everybody views of Facebook, Instagram, et cetera, but it's also the Android apps. It's all of the infrastructure code, all of Tupperware, any of our database, software, every service has it's in these various repositories. And because we have earlier thousands, but now tens of thousands of engineers working at a given time, we need to make sure that nobody steps on anybody's toes. And one of those ways is we have a relatively robust build system.  Before it lands, it has to be tested. So this is where we take, before what happened was we would have a local checkout on a machine of a relatively recent version of the repository, say it's a couple of hours old. Then what we would do is a Git checkout or a Mercurial checkout where I just use hard links into a specific directory for this patch that's being submitted. The patch will be applied, make sure that it applies. Then it would be built, make sure that it builds. Then the tests would be run, make sure the tests pass. All of this happens. And then that whole directory has to be removed. And these are hard links. So generally speaking, that's metadata operations for every file that you didn't touch. And then any file that was touched, obviously had to be removed. So the creation step was relatively quick because Mercurial and Git are pretty good about setting up their repos that are copies of each other. But the teardown was relatively slow, two to three minutes. In some of the cases, if you touched a core part of the infrastructure code, we have all of our, for example, we have all of our C++ libraries, Folly, for example, if you're touching anything in Folly, you're gonna end up with a lot of files changed. And those are like five to 10 minutes to remove those repositories. And while it's doing that, these boxes, there's only so many tasks and build processes these boxes could do at a given time. So if they're stuck doing removes for five to 10 minutes, that really kind of slows down. So like during peak hours of the day, it would take hours for individual code changes to land just because of this. So Butterfuss was looked at initially just for some of our more problematic repos. Android, I believe, was the first one. The idea being that instead of using the Git or Mercurial shallow copy, we would simply snapshot the repo, which is instantaneous. And then do the same build or whatever, and then delete the snapshot. Obviously, from the user point of view, the snapshot deletion is instantaneous. Obviously, it's really not because there's a lot of cleanup that has to be done, but the cleanup that needs to be done in the background is significantly faster than RM minus RFing and entire directory. So the repository cleanup, like I said here, was reduced by an average of 140 seconds. That's because that's about the average time for the Android at that time. It's 140 seconds and it went down to instantaneous. So it's reduced the capacity requirements for just that repo by a third. And the big thing inside of Facebook for services are their capacity requirements. So I mentioned before, we have a lot of machines. And so as programmers, we love to use every bit of resource that we have, but these resources cost money. And so you have to justify expanding out capacity. And so when we can help services reduce their capacity requirements, that translates into like dollar value savings. And that's the significant savings that can be passed like from our significant savings from our infrastructure budget. And so that was the initial use case. Obviously, once that turned out really well, it was expanded throughout the rest of the repos. Our build servers, it sounds like one kind of consistent thing, but each repository obviously has its own tricks and that sort of thing. The PHP stuff is different from the C++ stuff, which is different from the Android stuff and that sort of thing. So it kind of takes a little bit of retooling for each repo, but eventually, like this was a good enough initial result for them, that team to invest wholeheartedly in ButterFS. And now the entire build system is built on ButterFS snapshots for this reason. The next thing that was sought after was compression. Most of these are all tiny text files that are relatively easily compressed and ButterFS or Facebook spends a lot of money on flash. And we spend a lot of time and effort trying to make sure that flash lasts as long as possible. So compression not only helped us add more build jobs per server, but it also allowed us to contain the right amplification issues that we were seeing on flash and allowed us to have our flash, extend the lifetime of our flash, which again translates directly to cost savings, which is pretty impressive. This and this build service work was all done within Facebook, myself, Chris Mason, Omar Sandoval, other people work on the kernel team, we work on ButterFS, and we obviously help and we chip in other kernel areas, but we're the ButterFS developers. This work was done without our involvement and without really our direct knowledge. This was just a team that had a problem that they needed to solve, felt that ButterFS would solve it for them, went ahead and implemented it and told us afterwards. And I'm relatively pessimistic. File system developer tend to be pretty conservative. I was very pleasantly surprised at how well it worked that they had very little issues. Obviously, things can't come up, but generally speaking, they wrote the code and they rolled it out, worked great, they did it everywhere else, and we kind of like found out halfway through the process. Dev VMs, another big use case inside of Facebook. Every engineer inside Facebook has their own or multiple virtual machines in order to do their work. This is right now, we have 30,000 engineers, 40,000 engineers, so an unreasonable amount of engineers. So as you can tell, this kind of gets a bit unwieldy. Before we all had individual machines, can't really do that anymore. And honestly, a lot of the web engineers don't necessarily need a full machine, they really just need a box that runs HHVM in the most recent copy of the website. So one of the ways to address that capacity crunch was to implement Dev VMs. And these are VMs that are 800 gig VMs for the developers and they run, they're automatically set up, they already have all the repos loaded, they already have HHVM running, they have special sandboxes set up, they have access to their current website, the current website is installed on them every four hours or something like that. So they're having the entire Facebook code base installed on them to be displayed every few hours. And the entire Facebook code base is relatively large. So again, these VMs are backed by Flash, we wanted to be able to maximize this Flash usage, so compression was enabled to be able to maintain, allow developers to actually get their work done, because the other thing is 700, 800 gig VMs seems pretty big, but when you have all of the Facebook code and infrastructure that you need to do builds and that sort of thing, you start to run pretty close to full pretty quickly, which you would think would be problematic for Butterfest given our history with tight space. But it really didn't have too many issues with that, but compression was really, really attractive because it allowed users to not really bump up against that issue too much, because all this code compresses relatively well. And again, write amplification has turned out to be a really big issue with Flash inside of Facebook. And the less we write, the less write amplification, so compression kind of solves two pretty large problems, for us, in addition to allowing users to have kind of different access to the different Butterfest features, so snapshotting and that sort of stuff. It's allowed us to do different things for how we back up users data. It's not super interesting because generally speaking, users aren't going around nuking their VMs and having to restore, but it's an interesting use case. This was probably, this actually I feel like is the first coordinated effort inside of Facebook to have Butterfest as the root file system. And it went relatively well. Again, this is just something that happens. The only thing that I can remember with the dev VMs is again, you know, space related issues. You know, like I said, we have a lot of code and some service went wrong and started copying too many copies of the Facebook code base onto all users dev VMs. And this kind of put them in a tight bind and some of their boxes, some of them would fall over. So that kind of requires some manual intervention on our part and some code changes to Butterfest. But generally speaking, it's been relatively uneventful. Another use case, this is not currently in production yet. It's something that's being explored, but WhatsApp is reworking their offline storage. The way, because of the way that messages are sent on WhatsApp when the user is offline, those messages have to be stored for later. Currently, it goes into like these tiny little text files on XFS, they've been running into some weird scalability issues there. So in addition, not only with XFS, but with their older service. And so they've been rewriting their new service and they wanted to take advantage of the snapshotting features inside of Butterfest for ease of cleanup and removal as well as compression. Because again, these messages can be compressed relatively well. So this is, they've been evaluating Butterfest for this, which has been really interesting for us, because up until this point, Butterfest has not really been on a service where storage really mattered. So like, you know, starting a service or whatever, it's not super interesting from an iOS standpoint. And you kind of copy the service on, I'm sorry about the loads into cache. That's kind of it. The file system isn't really interesting there. And you know, the build services, like obviously they need to be able to read from desk relatively quickly. But again, once it's in cache, you don't really care. So from an iOS standpoint, these are not super interesting workloads. WhatsApp cares about read and write maintenance, for example. And so this is kind of the first major storage related service that's tried to evaluate Butterfest. So it's been going pretty well so far. They've run into some weird scalability issues inside of Butterfest. You know, you're talking about eight terabyte file systems that are mostly full. And because the files are sort of small, they're easily compressed and often end up in metadata. So, you know, of an eight terabyte volume, you're talking 800 gigs of metadata. So that just ends up with like, it's kind of interesting scalability issues. And also the way, because of how small the data is and how they clean it up, it's added a lot of, it's been a real stress on the allocator because of the fragmentation levels that it generates. And so thankfully we've been able to address these issues as they come up and it's been relatively smooth sailing. That being said, like we're still having issues and they still may not end up using Butterfest, but it's been an interesting test to see that so far it's worked relatively well. Because again, I'm relatively conservative and my view has been that if we care about latency at all, read and write latency, we use XFS because, you know, Butterfest is amazing, but it adds a little bit of overhead with, you know, online, with checksumming built in and that sort of thing that often store services don't care about because they have their own checksumming. So it's just extra overhead. And then finally our shared container pool. This is probably what we are most proud of if you go to like scale or any other conference where you hear talks given by our Tupperware team. This is the biggest infrastructure team within Facebook is the TW shared and Tupperware teams. They're the ones who handle containers and how everything works within such Facebook. So as I described earlier, originally this was with Loopback devices, Butterfest Loopback devices. Today it's much, much better. As we've validated and made sure that Butterfest is stable enough and fast enough to work for our systems, we invested the effort to reformat all of these boxes with Butterfest as the root drive and that's enabled us to do a lot of really interesting things and use some of Butterfest's features to really speed up and compliment our container strategy pretty well. So now instead of a Loopback device, the Butterfest images are composed with a base sub volume that is just a standard base sentos thing, which can be snapshotted then have the container packages applied to the snapshot and then these can be sent and received like the diff can be sent into a file. The file can be sent as our incremental feature where you can say, okay, give me a diff or give me this sub volume as a like an individual file that can be applied. And so these send images are generated which are significantly smaller than the actual service and then they are sent to the individual Tupperware boxes. They also have this base sentos image and so that's why we can just apply the diff. So it's just the service. So the snapshot is received. So the initial installation of the service requires significantly less IO than like copying the packages over and that sort of thing. That also allows us to do upgrades much easier because you're talking about just changing the bits that changed and not the entire image. So, and this also like this gives us the flexibility again to easily change different services. So the base image is the same everywhere and which means that if you want to tear down the old thing you just delete the sub volume, you snapshot the base image, you apply the new image and you just keep on going. In addition, we use compression again to reduce the right amplification in the IO load and space savings, that sort of thing. For our root devices, we tend to Facebook builds our own boxes. We don't use major hardware vendors which means for better or worse, we kind of for things like this, we kind of like cheap out for root devices which means we have to work around some really strange behaviors and ButterFS allows us to kind of get away with this with the compression and the send and receive. So we're just kind of not hitting these crappy disks that much and so it doesn't really affect services too much. In addition to the interesting features of ButterFS, we've also spent a lot of time getting it to work with C groups. There is a doing proper IO control with C groups in Linux has been relatively tricky because it's never just about the IO, it's about the entire path and there's a lot of weird gotchas that we ran into, I wrote IO latency, kind of the original IO controller for C group too and then Tage and wrote IO and so Tage and wrote IO cost which is the replacement IO latency was really brain dead simple, protect the main workload at all cost. You set a latency target, if it's latency target ever gets missed, we throttle everybody else and it's kind of like exposed a lot of issues within the entire file system and memory management and IO stack that had to be addressed but a lot of that work got done and then with Tage and its proportional IO controller, we're having even better IO handling but none of this stuff works with EXT4 at all. We've tried it with Exafest and there's still some issues there and we simply just not invested the time because generally in the places where we use Exafest, it's dedicated file system for a database, they don't need to IO share. So it's, ButterFS is nice for all that's other features but it's always the only thing that we've found that works with C groups. Not saying that EXT4 is off or bad or anything, it's just we've spent a lot of time making sure that ButterFS works and currently it's the only thing that does. So just kind of things that went well. These are just interesting tid bits that we've learned along the way of deploying ButterFS. Really, really early on we tried to use or what we did use ButterFS as our cluster backing store in some cases and we had this weird issue where ButterFS was complaining about checksum errors and again, I'm very conservative. If I see ButterFS throwing checksum errors, I just assume that ButterFS was broken. So we spent quite a bit of time trying to prove that ButterFS was wrong and when I say quite a bit of time like a week or whatever, I'm like really digging into this and then we started to discover that there was a pattern to the corruptions and it turned out that whenever there was a problem with the RAID controller, when you would reboot the box as it came up and probed to the partition table, it would rewrite the GPT header to the middle of the disk. And so when ButterFS came up and cluster started going and somebody went to read the file that had data in that area, the checksum wouldn't match and it would throw an IO error. This had been happening for years. These RAID controllers weren't only just used where ButterFS was, it was with XFS and it was clearly had been silently corrupting data and these cluster things were used for running AI tests, this is where big AI data sets existed and so they just had silently corrupt data and ButterFS was allowed us to notice it, get the firmware bug fixed across all of the fleet. And so that was a really interesting thing and that was the first time that I was like, oh, maybe we should probably trust ButterFS more than I do anyway. Chris has a very different view on this, but another thing that I think is really interesting another thing was we've found several micro architectural issues through our use of ButterFS. This is less like hooray for ButterFS on its own. It's more just because of the fast number of things that ButterFS does do, makes it pretty stressful on the CPUs with the checksum and the compression and the massive amount of off load threads that we have in order to handle all of this work means that we stress those CPU more than most and it provided us with a few pretty good solid reproducers to take to CPU vendors and get them to fix micro architectural issues and these issues had been plaguing us, again, we have a lot of machines so we see a lot of problems, a lot of problems and this kind of helped narrow down a problem and playing us for years where boxes would just lock up or hang out for no apparent reason or crash in interesting ways. ButterFS kind of helped us find that. Again, because I am relatively paranoid that ButterFS is gonna make the whole world burn down and it's gonna be my fault. I've spent a lot of time diving into what appear to be systemic issues with ButterFS. Because of the size of our fleets, we actively monitor every aspect of it and so every kernel message that ever gets emitted from any machine gets dumped into a centralized database that we monitor and things get pulled out so like ButterFS images, I have a little dashboard and I can go and say, okay, this is how many checksum errors that we've had all day or this is how many warnings and this is how many Enos based things that I've seen and so one of the things that have been bothering me was checksum errors and corruption issues and I've done several multi-week investigations into things, assuming that I was going to find a bug in ButterFS and so far it's hardware is worse than ButterFS, which has been interesting and been a good confidence boost. It's really kind of dispelled a lot of my paranoia around ButterFS and things that might be broken. That's not to say that I haven't been bugs, but it's mostly all of the things that seemed like really big problems that it's just like, wow, this is broken. No, it's been hardware issues or for example, with the ButterFS loopback devices, if you, because there were on any XD4 file system if the XD4 file system got full, it would return Enos base to the loopback device which would be translated as an IO error to ButterFS and then it would freak out and so there was a case where thousands of machines ran out of space on the XD4 and but we noticed because ButterFS started throwing thousands of warnings about corruptions and that sort of thing. It turned out to not be ButterFS, it was the underlying file system running out of space and just weird error handling stuff. And finally, like I've said on several of these cases, most of these were naturally developed by other teams inside Facebook. I have never once gone to a team and said, hey, will you please use ButterFS for this? Everybody kind of knows what ButterFS does. Obviously they know that we work on ButterFS and that's a big priority for us and so many teams have evaluated it and decided that, yes, this is what we want to build our service on and have gone and done it and we find out after the fact and not because of something goes wrong but because they start talking about it at performance reviews and it's like, oh wow, you did that and it worked. So it sounds like I'm negative about ButterFS but I'm honestly not, I just, yeah, I care deeply about the stability and success of ButterFS and I really hate it when people hit problems that are as a result of me making a mistake. So I want to be very confident that everything is right before people use it and so far I've been proved wrong over and over and over again. The, one of the funnier ones is I work on NBD for disaggregated storage services for a little while and part of that was using XFS over and that probably took me three to six months to stabilize just because it's a tricky thing. Most file systems are not really expecting your block device to just randomly disappear and with NBD that can happen and so there was quite a bit of time sunk into dealing with hangs and that sort of thing. I just recently found out a couple of months ago that some team had decided to use ButterFS on NBD and had been using it for two or three years and had no issues, which was just shocking to me considering how much time I had sunk into getting XFS to work well. Send and receive for containers was another thing. Send and receive, I know it works pretty well but I hadn't ever really considered using it in production yet because again, that's one of those things where if it goes wrong it's going to kind of go wrong in a weird way and then you want to be pretty confident it's working. Well, teams just decided to start using it and it's worked relatively well. And then with a compression, you know, I figured compression would be used for space saving issues, space saving things. I mean, that's kind of the main use case for it, obviously, but really it's been how we've solved our right amplification issues with all the flash and the fleet and this kind of translates to longer lives for our disks which translates to like real dollar savings. Problems that we've encountered, obviously, EnoSpace. I mean, it's not ButterFS if there's not an EnoSpace issue. That being said, it's been far less than I had expected. Again, this has been the thing that I've spent most of my career on unfortunately and it's gone pretty well. There's still some cases, there's some pretty bad cases that are left to be fixed that I'm working on but we're talking really rare occurrences in really bad extreme cases. So I've been relatively happy with that. Some different scalability issues. WhatsApp, like I said, has been pushing the file system to the limit in a way that we didn't really anticipate using ButterFS for ever. So those have been interesting to run down. They haven't been super difficult which has been nice, I've not yet run into a problem where it's like, well, ButterFS just can't do this. It's just been like, oh, we didn't anticipate this happening and get it fixed, so. And then there's just been loads of weird one-off issues. We've had a case where the checksum stuff was wrong. We've had weird cases with kind of weird corruptions that would happen. Happened to our, to Yen's Expo, the block maintainer for the Linux kernel. He had a pretty kind of work load that tickled it. So it's mostly just affected him but we fixed it anyway, we like him, so. But nothing really that stands out.  Things that are left to solve. Queue groups, quota groups inside ButterFS still, obviously we would love to use this for containers to be able to like fully contain space and that sort of thing for stacking services. The overhead for key groups currently is still just too much and frankly it doesn't, it breaks in a lot of weird cases. It's not something we spend a lot of time on, so I'm hoping to get to that in the next six to 12 months to really nail that down. Omar Sandoval is working on a subvolume encryption and hopefully that'll be in the next six months and that'll be really interesting. Again, for containerization workloads, we want to be able to encrypt containers and be really certain that we can stack different services with different privacy requirements safely. Balancing space in extreme cases, the way, this is just an extension of the InnoSpace stuff, I by and large, everything works fine. It's just weird extreme edge cases that are kind of problematic. So this is kind of another big focus of mine. And then RAID in five and six. The RAID striping and mirroring and striped mirroring works relatively well. It's pretty solid. Five and six, still there's a lot of edge cases and just more testing needs to be done. So that's another big work area for us next year to two years. Awesome. Well, that's it for me. If you guys have any questions, I'll be happy to answer them. Thanks. Okay, cool. So we have a few questions here. I'll just kind of run through them. So Nikolai asked, there are a lot of people claiming that ButterFS is not fully stable slash functional. What can you say about this? I mean, everybody has their own experiences, right? There's no doubt that ButterFS has had issues previously.  And, you know, we don't take failures lightly. Obviously, you know, Facebook is far more fault tolerant than your average user. And so if something goes too wrong, right, we just reprovision and keep going. And, you know, a random Linux user is gonna have a very different view on those sort of failures, right? That being said, any sort of large problem or kind of systemic issues that you're gonna see would affect us in production. And we can't really afford those. We can't afford to have engineers running around fixing things all the time. So by and large, for Facebook and our use cases, which cover a large chunk of most people's use cases, it's been extremely stable and functional for us. Nikolai asked, how exactly does compression side solve write amplification? So this is probably more to do with Facebook's utilization of the flash. We obviously log a lot of things per disk. And so there's a lot kind of those constant stream of text being written to the disk just from logging. And so compression kind of handled this, you know, with the X-D4, we saw a significant amount of write amplification and the compression handles this by, you know, compressing the data and making sure we're not writing out as much. And the same goes for like file updates and that sort of thing. We do, like I mentioned in the talk, we push our entire website to our web services boxes pretty consistently four to eight times a day. And you're talking about 20 to 40 gigs worth of writes 48 times a day. And so compressing this really helps with how much we're writing, where we're actually hitting the disk with, which has drastically reduced our write amplification issues that we've seen in the fleet. Nikolai also asks, what compression algorithm do you use? I believe it's ZSTD. Well, yeah, I'd have to double check. I think it's whatever the newest one is that we added. We, one of the Facebook actually employs a guy who does compression algorithms and he wrote one and got it into the kernel and then we added that ability to ButterFS. And so that's what we use. How much, we also asked, how much space does that save for the repository clans? I wouldn't know. It's kind of hard to say exactly. I could probably go check, but off the top of my head, I have no idea. Oh, Chris just messaged me. We are in date using ZSTD. I'm sorry, I'm gonna butcher your name. Can you share performance numbers of ButterFS? I don't have any off the top of my head. So Facebook is really weird when it comes to performance. Every service kind of has its own requirements. I can say, and other services are better about testing their performance requirements than others. I can say for the web tier, it's extremely latency sensitive. Like I said, we love a lot of things and so the IO path tends to be pretty hot, weirdly, for the web tier. And so much so that latencies result in lower requests per second for the web tier. And web tier is kind of the first proving round after the build servers of will this work? And we consistently saw higher requests per seconds on ButterFS than EXT4 because we had generally speaking lower latencies because of the compression because we're not writing out as much. We were trading CPU for IO costs and with these kind of consumer grade solid state drives that ends up being a pretty good speed up for us. That being said, for what's that? For example, they're currently evaluating us. We see maximum read latencies on their disks at around 300 to 400 milliseconds, whereas EXFS kind of sees 50 milliseconds, maximum latencies, obviously the averages are lower. So ButterFS's averages are always significantly lower. So EXFS is around what I want to say like 10 milliseconds whereas ButterFS is around two to five. But again, for capacity sake, we care mostly about maximum latencies and so that ButterFS is for sure a lot more sawtoothy here. And so it's something we have to dig into. What is the future of ButterFS over databases? We will probably never, I won't say never, but we have no plans to move our MySQL databases over. That being said, we have a lot of services that use our LuxDB database. And that works pretty well because it's continuously appending and that right pattern works really well for ButterFS. So we have services to use them. If you use Docker, do you also use Docker as ButterFS storage driver? We do not use Docker, we have our own home ground system. Think we're just about out of time here. Let's see, is there a lower limit in terms of the use cases for ButterFS? For example, does it make sense on a small server or a high end desktop 32 CPUs, nine terabytes of HDD STD space? I wouldn't say there's a lower limit. Well, perhaps the, you know, if you start, we have machines that use atoms like atom CPUs and ButterFS is kind of CPU heavy just because of all our worker threads, especially if you start adding in compression. So, but generally speaking, you don't have to have a beefy box to run ButterFS. Anyway, I think I'm out of time. So thanks everybody. There's a Slack channel that you can ask questions and the hashtag to track Linux systems.


Hi everyone, Joe for Jaspy's casebreaks.com coming at you with 2023 Bowman Baseball It's a blaster box big random team number 25 with eight of those baseball mixer spots being given away That right there that looks exciting. It's gonna come up in the next video But first let's do the break itself big thanks to this group for making it happen. There are the teams right here Let's roll it. Let's randomize names and teams five and a five ten the hard way one two different dice for for the giveaway part as always four five six seven and Then ten the hard way for the teams After ten we got the Braves down to the diamond bags All right, so here's the no trade windows as you know by now Here's the first half of the list and there is the second half of the list right there. All right, let's just print and rip That's right. Trev. There's anything like their regular big box products. It should be nice Jeremy. What's going on? How are you? Happy Tuesday Yeah, so I don't know what this You know the the jazzy's exclusive baseball big box edition What is that? What's that? I don't know. I don't even know what the difference is But we're gonna find out I think generally speaking from more knowledge of our What people like to buy in the store Single cards people like to buy in the store what people like to buy and chase in our group breaks What people comment about on our social media we just take all that information and we go a hit parade What do you uh? We think we got this set of players We kind of want this kind of price point our crew likes this sort of stuff these sort of players What can you do for us?  Hopefully people will like it All right, so I think for the purposes of this Filler break usually paper or commons won't you had no paper will ship Although I will ship that uh drew jones Bowman first paper for Eugene in the diamond backs And oh nice Bowman scouts top 100 number nine Jackson holiday I think I saw a recent Prospect report that has him in the top five now That's 83 out of 99 Nice trev met the hippery people at a show in Kansas City During the nfl draft brought a whole bunch of body about a whole bunch downtown That's nice Yeah, there we've uh, I've met him a couple. I always There's always a team of two Of the main guys. I always mix up their names I'm terrible with that sort of stuff. So apologize in advance if they're listening, but um But yeah, they're really nice though super friendly very knowledgeable about the hobby, you know Their temperament I think a lot like the jasps temperament. They don't get too high about stuff They don't get too low about stuff. They don't Don't really make knee-jerk reactions on a lot of stuff You know Pretty pretty good knowledge of the uh Of the hobby to do a good job. I think Uh jason you got the you got that uh refractor With the red socks. All right. Now here is the big moment here Let's see who's going to win those spots in that baseball mixer. Let's gather everybody's names one through 30 And it'll be the top eight Let's get everyone's names in this clean list first. It'll be the top eight after After seven top eight after seven. Good luck one two three four five six And seventh and final time So from nine on down it's going to be sad times Peter My apologies sad times Not only for you You're very close. No cigar, but for everybody nine through 30, but I appreciate you giving your shot Thankfully, there is another mixer and another couple fillers attached to that mixer. We're giving away spots Jaspy's case breaks.com. So get after if you want to try again top eight. Congrats to you Happy times for Phil Andrew Andrew Eugene Jared Jeremy Brian and Andrew Happy times for the top eight. We'll see in the next video for the break itself. Jaspy's case breaks.com. Bye. Bye


Welcome, everybody, to another episode of our Friday Ask and Answer with Fundraising Academy. We are so excited that you are here with us today. We want to make sure to express our gratitude to all of our sponsors who've been with us many from the very, very beginning. Boomerang, your part-time controller, Fundraising Academy, non-profit nerd, the American Non-profit Academy, non-profit Atlas, non-profit Thought Leader, and staffing boutique. These are the organizations that are with us day in and day out and allow us to share now more than 500 episodes of the non-profit show. Without them, we would not be here. So thank you, thank you, thank you. If you've missed any of the episodes or maybe you want to share something that you've heard, you can find our robust archives on Roku, YouTube TV, Amazon Fire TV, and Vimeo. We can simply speak into your smartphone and voila, there we will be. And yet it gets better. We are now putting all of our episodes onto a podcast format. We're streaming in more than 20, I think it's over up to 22 platforms now. And again, you can queue us up, subscribe, and then we'll come to you every day, which is really an exciting way. If you'd like to consume your information, perhaps while you're commuting, working out. Doing housework, I don't know, whatever, but get going with us on our podcast. Okay, deep breath. I've set all my housekeeping rules, set the stage. Toni Bell, yay team, welcome back. Thank you so much. Well, you know how much I love being here with you and answering questions from your viewers and subscribers. There's always really provoking questions that lead to great conversation. So yes, thank you for inviting me back. Well, you are great. You know, I think, God, Jared and I say this all the time. If I've had your wisdom 30 years ago, even just a glimmer of it, I could have raised so much more money from my community. I've always raised money. I've always been out as a volunteer, a board leader, but I never had all of this wisdom and this process. That's very simple. It's very natural that Fundraising Academy is really educating fundraisers about. And so it's been remarkable to see how you and your team continue to share this information. And I like to think the next wave of leadership is gonna be so much better than I ever was. So our communities are gonna be better, you know? Yeah, and that's certainly the hope, Julia, through this curriculum is that fundraisers are raising more money to better support the communities that they serve and total transparency. I didn't have this wisdom either until I got introduced to this curriculum. So I'm just one of many that have the opportunity to be great ambassadors and messengers of this great cause-selling curriculum. You know, we've done a lot of different types of programming with you and various members of your team and there's so much more. If you go onto the fundraisingacademy.org website, you'll find tremendous information that's most of it's free. You can share it with your teams. It comes in so many formats. But I would really encourage you to take a look at this. Even seasoned fundraisers, there's something new and something valuable to learn. I really do love it. And we love your partnership with us. We're always so enchanted by the things you say. So are you ready, my friend? Yes, I am. I'm really looking forward to these questions. Okay, Manny from San Diego, California writes, have you ever heard of a board policy that prohibits husbands and wives or family members from serving at the same time? We've had a struggle getting board members and have an opportunity to get the involvement of one of our current board members' husbands to serve. Yeah, well, thank you, Manny. I think that's a great question. And it's interesting. I have a little experience with this, with an organization that I worked with about, gosh, 12 or 13 years ago. But the difference, however, is that with that particular organization, we had a board application for couples to serve on the board together. And together they would equal one vote. And part of the reason that we did that was because for this particular organization, its founder got it off the ground through, like many folks do, their personal network. And there were just a lot of couples that really gravitated towards the mission. And so when we got to the point where we were ready to actually incorporate and populate a board of directors, a lot of the, you know, if we took all of the couples, we would have had, you know, 40 board members. So what we did and also recognizing that a lot of times it's difficult to get board members to meetings. Sometimes it can be difficult to have quorum. So by accepting them as a couple, you almost always had quorum. But again, if they came to a meeting together, they still represented one vote. Okay, now I have never heard of that. Have you, had you, you said before you hadn't heard of it, but have you heard of it since or do you think this is something that could be adopted? Well, I don't know that I've heard of anyone else using that particular model. But to Manny's question, I'm not aware of anything that prohibits, you know, husbands and wives or family members from, you know, from serving on the board. Certainly a lot of grassroots organizations when they start and first incorporate, they will incorporate with family members because they want to incorporate with folks that they know and trust to initially get the organization off the ground. So I don't think it's uncommon that there, you know, that there are family members serving, you know, on the board at the same time. You know, I would think this speaks to your conflict of interest policy and really going to your bylaws to see if that is in there. Because if it hasn't been put in there, it doesn't mean it can't be changed. Right. But you need to take a look at that because I know there are organizations that are really strict about this even to the point where, you know, they're not allowed to have an employee relationship on staff. So you couldn't be like the parent or sibling or spouse or somebody that works at the organization.  And then as a team, Manny, figure out what you're going to want. But pardon me, filling boards. This is a, I don't know about you, Tony, but I'm getting a lot more questions and I'm hearing a lot more about this. Like, hey, we're struggling. We're struggling to get people to volunteer to fill these positions. Well, and the board experience, like many experiences right now in this landscape is very different. So, you know, so it's harder. You know, it's a little bit harder to prospect. We all know that even for donors, it's a little bit harder to prospect, you know, and this, you know, in this particular landscape. So I can understand why, you know, recruiting and retaining board members right now can be a challenge. But you brought up the perfect point there, Julia. And that is, you know, how does your bylaws, how do they or do they not support this? And just making sure that, as you said, in your code of ethics and within your bylaws, it's just clearly defined how that structure works. And again, I think, you know, you can make these changes. You can modify these documents, but you got to modify the documents. You can't just say, oh yeah, this sounds like a good idea. I mean, you kind of got to take those steps, right? For sure, absolutely, we absolutely have to do that. Well, Manny, we probably gave you a lot more to think about than you wanted with this question. I think he was probably hoping, is this a yes or no? No, not really. Okay, Rachel from Lexington, Kentucky writes, we have a fairly large community room, event space in our building. Recently, someone approached us about running some classes in the space. They would rent from us and it could bring in a small bit of revenue. Any thoughts or things that we should look for before we make a commitment? Well, one, Rachel, congratulations on thinking about, creative ways to diversify your funding streams, right? And if you have the luxury, and I say luxury, because not all organizations have the luxury of being in a brick and mortar space and certainly not a brick and mortar that would afford them a large community room, I think it's really smart to look at how you might be able to get some financial benefit from that space. I am not a leasing agent or a leasing attorney, right, but there are a lot of things that immediately come to mind when you start thinking about going down this road. Because in a lot of cases, you're going to be leasing to individuals that you don't really know. So, right off the, you need a leasing agreement, some type of rental agreement. Do you have the appropriate liability insurance to host events? What does the food and beverage kind of contract or restrictions look like? How does it impact or not the way in which your organization files its annual taxes? So, that might be the first place that you wanna look at is talking to your accountant around how does this type of revenue affect the way in which we submit our taxes? So there's a lot of things into it and then making sure you have the resources and the tools necessary to effectively reserve the space and manage the reservation process. But, so I think if you think about all of those things and what the potential earnings might be, compared to what you're gonna need to invest in upfront in order to make it sustainable and to help you grow it, then again, I think it's a really, really smart opportunity and something really worth investigating and doing your need discovery with. But I really support it if you have the space in your facility to do it. You know, I love all these points and you're right. I mean, because the cost of doing these things, it's not just a benevolence issue, it really has hard costs. And I also think there's something magical about getting new faces and new people to your campus or to your site. You never know. I think how hard it is in our lives to get people that we think are interested in us to come to our campuses and our offices, right? Well, yes, I'm sorry, I didn't mean to interrupt you. No, no, but I'm just thinking. It's like, holy moly, maybe this is something that one in a hundred people, I don't know. But I mean, think about it, that you could introduce your work to. Well, I love that you brought that up. I have the pleasure of having office space within an organization here in Broward County called Artserve, which is an arts incubator. And it's a really large facility and they do a lot of rentals. So of course, one of the things that they do when they rent the space is they welcome everyone. So pretty much every event, especially some of the smaller convenings, the CEO or the development officer or someone from the organization before the workshop or whatever it is starts, they will come in, welcome everyone, let them know where the restrooms are, any other kind of facility protocols and then a quick little elevator speech about the organization and what it does. Okay, yay team, now that to me, that's the gem of all those things that you said because that makes for real connections. And if nothing else, maybe even additional leasing opportunities, right? Absolutely, without a doubt. I mean, because these people that are gonna be coming to something like this, most likely would be those people involved in other community events where they'd be like, where do we meet? Now that we're kind of starting to come back out. Super cool. Well, Rachel, you heard from the master. I think these are some really cool ideas. Shonda from Cincinnati, Ohio writes, Tony, have you ever heard of development teams holding joint session trainings or meetings? We are thinking about bringing in a consultant who would be great, but they are expensive. We're wondering if we should join up with another nonprofit and share the fee, which would make it easier on both of us. So we're getting some really great questions today. We are, we are. We're getting some really, really good, thoughtful, creative kind of, you know, just creative questions. So Shonda, have I heard of this specifically? No, do I support it? Absolutely. You do. I do support it. I think, well, so when we think about the other ways in which development teams come together in training, we would think about AFP workshops or our fundraising academy cohort, which is comprised of representatives from development offices across the country. So I don't, you know, so I definitely don't think there's anything to worry about or, you know, to be concerned about and collaborating with other development teams in your area to share the cost, to have some type of professional developments. I absolutely support it and think it's a super idea, especially recognizing, you know, the how little money is usually allocated in it, you know, in an annual budget for any kind of training. So if you can share the expense and have the shared experience, which I think is going to just increase the overall ROI experience, right? You know, the overall experience for everyone by having more folks with different opinions and varying ways in which they look at the work that they do. So yeah, I think it's a super idea. Now, let me ask you this, Tony, would you try and get somebody with the same mission or similar mission or would you specifically look for somebody with a completely different mission? So for example, a cultural group and a pet rescue group, or would you try and get together? Yeah, no, I think that that's really smart, Julia. And I think it's great that you brought that up. Yes, I would absolutely look for an organization with a completely mission, you know, a separate mission from not even closely related. Because again, when you have those conversations with folks that don't live in your space, it's remarkable the things that you'll hear because they're not living in with the same kind of sense of confinement. And I'm not sure if I'm saying that, you know, the way that I really want to communicate it, but there's a lot of freedom in having conversations with folks outside of your space. They see things a little differently. And I immediately think when the example that you gave an arts organization and a pet rescue, well, there are lots of ways that those two organizations can collaborate. Yeah. Lots of ways that they can find to collaborate. And probably find lots of commonalities amongst the, you know, some of the donors that they have and some of the donor personas that they go after. There's probably lots of similarities. Interesting. Okay. So then let me dig down a little deeper. And I know you travel throughout, you know, the country, you came to my state not too long ago to deliver an address in another part of the state. But, you know, when you look at something like this, would you recommend to Shonda to, she uses the word joint training, would you recommend that maybe she goes out and she gets a couple or several different groups? Or do you think it's smaller? Do you think it's better to have a smaller, like, two kind of group thing? Yes. I mean, that's a good question. I think my answer to that would be based on the level in which the Shonda, for example, if Shonda is going to be the leader of this initiative, the amount of time in which Shonda wants to invest in this, right?  And I just say that in terms of logistics and payments and invitations and all of that. I mean, there's certainly a lot of value in the conversations that would take place. And again, all of those multiple voices, but just the logistics. So my recommendation would be test the waters with just one organization initially. And then see how that goes. Shonda needs to know Rachel from Lexington because Rachel's got the community room, right? The question before was about having a community room. But yeah, I think you're right. I like what you're saying about, you know, trying it out, seeing how it works, you know, you and I working with public speaking, it's quite a bit different when you do a seminar kind of style work versus a ballroom of people. Right. You know, it changes. Right. Yeah. And, you know, and there's lots of creative ways for Shonda to think about how to introduce this and implement it. So if, for example, if there's concern about making that big investment in, you know, a consultant, then perhaps they lean into some of the recorded episodes of the nonprofit show or some of the webinars pre-recorded on the fundraising academy website. And maybe they test the waters by everyone watches a nonprofit show or a webinar on a particular topic and then come together either in a room together or virtually and have a conversation around what they saw and what were their takeaways and, you know, and kind of see what the synergy is like between the two organizations in that kind of professional learning environment before you make the larger investment and take the deeper dive. Right. I love that. I think that's really smart. You know, but I think the starting point for me was you embracing this concept and saying, yeah, put it together. And then to your point, try it out before you really make a big gathering because yeah, that's cool. Well, I hope that, yeah. Yeah, no, I just, you know, collaboration and this is just a collaboration collective impact kind of, you know, opportunity which I always support completely. Okay, let's go to Francis and Cordelaine, Idaho. What is the average number of members for a nonprofit board to have? I'm tasked with looking at this and I have no idea where to begin. So I, you know, it's a great question. The solid answer is 15. So 15 is kind of the average number of board members, 13 to 15 for organizations. I think, you know, first you need to take a look at what are your expectations of the board? How are they meant to serve the organization? And that will help you, I think, better understand the number of board members, you know, that you need and the types of board members. But the simple answer is that 15 is kind of the average. When I was consulting with organizations, I would always say like nine to 13 was, you know, was kind of the sweet spot for that. But really Francis, you know, give and not that you haven't, I'm sure that you have but give a lot of thought to, what are your expectations of the board? How are, you know, what is the intent of their service to the organization? And that I think will help you determine what is the right number.  So you're probably looking at like regional, you know, board members or folks that don't necessarily live there or can you manage that? Can you manage people that are coming from the region? And, you know, when you look at communities like this, especially in the West where we get so spread out, you could easily be a one to two to three hour drive to get to the meeting. So, you know, if you're physically wanting them there versus yeah, that's a lot to think about. And I appreciate you giving us a number. Obviously the number should be odd, correct? Correct, yes. The number should be odd. And something else came to mind too. And, you know, when we talk about the average number of board members, I mean, I'm making the assumption that Francis is with a 501C3. But Francis may be with a 501C6, a membership organization where typically they will have boards up to 26 to 35 individuals because membership is such a critical part of their organizational sustainability and growth that they want a large, you know, a larger board, you know, like a Chamber of Commerce, for example, wants a larger board with representatives of all of the different industries and markets. So again, you know, my answer just kind of naturally came from the lens of a 501C3, but it may be a C6. Yeah. You know, and then the answer is a little bit different. Yeah, very smart. I appreciate you tagging us to that. Okay, now you know, Tony, how I love my name withheld questions. Okay, name withheld Salt Lake City, Utah. My current nonprofit has never used business cards and I think it is an important tool. When I asked for them, there was a bit of a pushback. What should I do? I could get them myself, but I don't want to cause a ruckus. Okay, ruckus. Well, thank you for this question. And it's, you know, so my first reaction would be that in this particular landscape where so much of what we're doing seems to be remote, I could see and understand why an organization might give some pushback for that type of investment, right, if, you know, you're counting every penny, every dollar counts, and so, you know, I might understand that, but, you know, but name withheld is saying has never used business cards. So that kind of sets a different kind of story and makes me look at this a little differently. It's my opinion that at a minimum kind of marketing 101 that an organization should have letterhead, envelopes, and business cards. Yeah, yeah. That's just, and, you know, and name withheld, if you need kind of proof or documentation, I always turn to Google and I'm sure that there's something on Google and I know that that's kind of a funny way to answer this, but if you're looking for like, you know, some type of case statement or some supporting documentation, I'm sure that there's something there, but I have always, I would always support the notion that at a minimum, your organization should have branded letterhead, branded envelopes, and branded business cards. Yeah, I agree, and I think that, you know, I don't know what this person is doing, but especially if they're in development, you know, handwritten notes and things that go out, those business cards need to be put into those pieces going out because you're right, it's a marketing tool, you know, it's a communication tool undoubtedly, but certainly a marketing tool. So yeah, I would say that kind of makes me think they're not making investments where they need to. And you bring up an interesting point, Julie, but I think regardless of your role within an organization, you should have a business card. Okay, yeah, yeah. I really do, I think, you know, it's just, it's that value of being part of the organization, it's, and again, as everyone serves as brand ambassadors for the organization. Right, right, it's important. Well, you have been, you know, you always make me think, you always give us new ideas, we really, really, really love it when we can get you away from your busy, busy, busy schedule. I know that you move throughout this nation dealing with a lot of different people, Fundraising Academy being a part of National University, you have a lot of responsibility and you bring to us a tremendous amount of knowledge, but you are doing a lot of big, big thinking things. So I appreciate that you like pull it down a little bit for us and bring us your wisdom. Before we let you go, do you have anything that's like really fun and exciting that's going on forward, you know, that you can share with us?  No, no, all good. So for anyone that's, you know, that's joining us, that is going to be at AFP icon, we are not exhibiting a booth this year, but we have a couple of leadership tracks and workshops that'll be facilitated, you know, I'll be there, our director of the Fundraising Academy, Pearl will be there. So please, we'd love to meet folks and get together for a coffee. We are right now launching our online Fundraising Academy portal. So really excited if you go to Fundraising, if you go to the website, fundraising-academy.org, you'll find links to the live portal. So we're really excited about all of the professional development that's available there for folks. So thank you for the opportunity to share all of that. But yeah, it's an exciting time for the Fundraising Academy. Well, we need you. We need you in our sector and in our corner. I don't care how seasoned of a fundraiser you are or how confident. There's so much that you share with us. It's just remarkable. And I always love going on there and seeing what you've got. So Tony, thank you and thank you to your team. I'll shout out publicly, you know, they are lovely people to work with. We work with them more than we work with you and they are fabulous. I mean, from all angles. So I just wanna kind of give you a shout out. Again, we wanna thank all of our presenting sponsors without them, we would not be here. Umarang, your part-time controller, American Nonprofit Academy, Nonprofit Nerd, Fundraising Academy, Staffing Boutique, Nonprofit Thought Leader and the Nonprofit Atlas. Hey, Tony, I hope you get to have a restful weekend. You as well. Always a pleasure to see you and participate in these conversations. It's such an honor. Well, you are my hero and the hero of many other people. I'll say this before we sign off. I was at an event several weeks ago and somebody quoted you from having watched you on the Nonprofit Show and they're like, well, you know, Tony Biel says, Tony Bell, excuse me, Tony Bell says. And I was like, holy moly, that was pretty cool. Well, that's pretty humbling and thank you for sharing that. It was fun. Hey everybody, we like to end every episode and as we're ending our week, we want to remind everyone and ourselves, stay well, so you can do well. We'll see you next time, everyone.


everybody forced here ripping case number two of 2022 dynasty f1 random serial number left side serial numbers we're going to random right now to see who's got which number and then we're just going to get right into the case right after that good luck everybody three times three times on each names first three times one two three nick and down to christopher l and then three times on the numbers here one two and three seven down to five all right nick end with seven and three dale c 10 sunghyun h with nine amod eight ck with two danie c six mark d four simon s one and christopher l five i'm gonna post those right now i'll get you a link i'll also call out who's hit is who's as we go along here there you go and again f1 mixture spot still available for uh this morning really got that early start got that tops release they get you get started at 10 a.m. eastern like that good luck y'all come on come on all right okay starting us off oh is that is that our city's color 10 of 10 so 10 spot here here you go 10 spot with dale daniel rickardo patch auto two color patch auto daniel rickardo 10 of 10 all right solid start i'll see some uh see some let's see some crazy let's see something crazy this one big time logo patch or something i'm just gonna start ripping the bandit off here what's up josh well it is a mix shoe mucker nine of 10 two color patch auto the high numbers hit sunghyun h with nine so we have a nine hit and a 10 hit so far right there we go the high numbers dreaming big right now is this gonna be lower number now oh okay this one this one could be pretty nice i'm just gonna just gonna go for it two of five christian horner a nice little red bull uh racing patch auto right there christian horner two of five going to ckl ck with the two spots red bull racing patch let's get back let's get back with the drivers now seven of ten kevin magnuson dual patch auto seven spot nick n n nick n seven spot kevin magnuson all right well let's see a monster in the last one let's see a monster in the last one our lower our lower number it hit and this one was uh was a principle so let's see a big time low number driver or just a nasty driver luham or max or something like that joe rookie and we've got a terry botas six of ten triple patch auto six of ten six spot danie c sick botas oh yeah multicolors here black and white red white red black all right there we go that'll do it for case two everybody case three is coming up in just a couple minutes here thank you thank you


Fy ydych chi'n bwysig fod yn ddweud y bwysig, ychydig yn gwybod yn ddweud y siwer a'r effaith yn bwysig? Mae'r ffordd yn bwysig i'r ddweud a'r iawn i'r ffordd yn cynghorol sy'n gweithio'r rhaglen y bwysig. Mae'r bwysig yn ddweud? Mae'r bwysig yn bwysig sy'n gweithio'r rhaglen y bwysig? So, the children and the poor, so we've got it up, right? Yes. Yes. Well, though, in single matter, if they were poor, they would be helpless. Do you have children? Yes. No, look, children have, you know, their parents, in a sense, hold their rights in... Trust? Yeah, they have trust in a sense. Children don't have. They don't embody the four rights because they're not fully rational in that sense. They're not capable of dealing with the world, and parents, that's their responsibility. Responsibility is to help them get to the point where they are, you know, where they're adults and they take care of themselves. In a case like that, in an objective society, I have no doubt that a charitable entity would come in and help those kids. And take care of them. I think it would ultimately be up to private charities to take care of the poor, of the helpless. You know, you can imagine something born of a poor, clean kid. You know, just can't take care of yourself. Then it's their family that'll take care of them, or it'll be charity that'll take care of them. What isn't legitimate, what isn't legitimate is that I be forced to take care of them. I can be approached and, you know, I can be reasoned with. I can be, you know, people can ask me to help them. But they have no right to pull out a gun and force me out. And as soon as you have government doing that, then you're basically... The only role the government has in getting it to defend it is right. If the parents are using the kids, then government has a role to step in and stop at abuse. Because right to be violent and trust that the parents are holding is being violent. But if the parents, if the kids run away from home, if they're just poor, if they're just... Dad is an issue for charity, it's not an issue for the state. I'm supposing you had somebody who'd support themselves because of someone... They couldn't get charity and nobody would have helped them. Then they would have. But I don't think that would happen. I don't think there's any evidence in American history. Or even in the history of other places that have approached freedom. Everybody's just approached it, they've never attained it. Even in a place like Hong Kong, as brutal a little bit of a place it is, as poor a place it was, at least for some people, but free, people weren't dying in the streets. People taking care of it. They were charities even in the poorest of places. There was no welfare state in Hong Kong for many, many years. And they had people by their millions, by the hundreds of thousands of people. Emigrated in, right? People were escaping to get the welfare of other countries. People were coming in. I mean, that to me should indicate something. In every free country in the history, to the extent that they are free, people want to move it. People are not climbing. I mean, some people want to come into the welfare state. But people certainly are not climbing with climbing to go to the Soviet Union. People are climbing to go to North Korea. They won't ever go into Europe.  The degree to which people want to move it. People want to go to Japan even though the Japanese won't let them in. People want to go to South Korea. People want to come to America. They want to come to the UK. They don't want to go to North Korea. They don't want to go to Iraq. Arabs want to come to Israel. They don't want to go to West Bank. They don't want to go to Jordan. But if Israel opened up its job market to Arabs from all over the Middle East, it could attract millions and millions of Arabs to come and work in Israel. Because if Arabs in Israel are freer than they are in any other country in the Middle East. And indeed, and this is not what you're talking about today, but indeed, just the historical fact. Between 1890 and 1948 when the State of Israel was established, the Palestinian, so-called Palestinian, the Arab population of Israel grew dramatically. Not because of both rates, but because of immigration. From Syria, from Lebanon, from Jordan, and from Egypt, and from Iraq, and from everywhere. Why? Because those nasty Jews were building industries, they were building businesses, they were building roads, they were creating civilization, they were creating activity. And all these Arabs wanted jobs. So they can't, all the Palestinian problem, the so-called Palestinian problem that exists today, is all the fault of the Jews for building up a 73 countries to begin with. Because those Palestinians could have been slaves in Syria and Jordan and anywhere else today, and they wouldn't be so-called refugees in Palestine. But capitalism, freedom, individualism, creates prosperity. And so prosperity, again, is a proxy for freedom. And with people that do it poor, I can guarantee you they're also unfree. Recallation, travel a little bit around the world that you can see. You done? Thank you. It's always a privilege and a very, very stimulating experience to hear and speak. I do want to wrap up quickly because we have to fill this room by 9.30, so we've got 45 minutes for drinks. Just before we do that, we would like to present to you your own with a copy of Douglas' book, which he's kindly signed for us at the start of the evening. So thank you very much. Thank you to you on what's more. So thank you again Douglas, thank you again to all of you. Drinks are in the room behind you. There's always been a room by 9.30, you go by. Thanks very much.