Okay Hi everyone Thanks very much for choosing to give me your attention for half an hour of your finite lives I'm the good Tom Stewart. I'm gonna tell you about Monads and how we can sort of uncover them by refactoring our Ruby code So the ideas in this talk might be unfamiliar to some of you Before we get stuck into the details. I want to just warm our brains up with something simple before I get into the difficult bit People always use analogies to explain Monads It doesn't help so there'll be no burritos or spacesuits or elephants in this talk But I do want to talk about some related ideas not analogies But just related ideas that will put your brain in a receptive state for the stuff. I want to explain to you I'll start with a rhetorical question. What is a stack? Well It's a kind of value with certain operations and Those operations are called push pop top and empty. I'm talking about an immutable stack here So push and pop don't mutate the stack. They just return a new one You have to use top if you want to look at the top element We also need a class method called empty which creates an empty stack to get us started For a value to qualify as a stack its operations have to follow certain rules For stacks the rules just say that the operations do what you intuitively expect when you push a value onto a stack That becomes the top value Pushing then popping is a no-op an empty stack is empty and a stack with something pushed on to it isn't empty We can implement the stack operations however we like Here's a class called a raise stack that implements the operations in a particular way It stores the stack contents as an array of values push adds a value onto the front of the array and pop removes the arrays first item This class as linked list stack implements the same operations in a different way It stores the top element and a pointer to the rest of the stack So top and pop are just attributes and push stores its value in a new stack instance pointing at the old one We can use the stack operations without knowing how they're implemented if we make an empty array stack and Push two values onto it and then pop one of them and ask for the top value We get the result that we expect if we do the same thing with a linked list stack It works in exactly the same way We can define more operations in terms of the old ones So for example, here's a method called size that recursively calculates the size of a stack by counting How many times it has to be popped until it's empty? So as long as the stacks got working implementations of empty and pop we can implement this size method on top of them It doesn't matter what stack implementation we use size always works the same way an empty array stack has size 0 if We push two values on its size is 2 an empty linked list stack has size 0 And if we push two values on its size is 2 So what do we mean by stack? It's really a specification an Implementation of a stack provides certain operations that follow certain rules There are two benefits of that firstly those operations provide a sort of common interface across many possible implementations and Secondly they allow us to build shared functionality like that size method that works with any implementation Here's another question. What's the collection or at least in Ruby? What's the collection? Well, it's a kind of value with certain operations actually just one operation in Ruby called each and That operation follows certain rules actually just one rule in Ruby Which is that each calls a block with a value zero or more times in immediate sequence? That's it Now we can implement that operation however we like Here's a hard coded collection whose each method literally calls a block with one and then with two all the way up to five Here's a generated collection whose each method calculates those values dynamically in a loop and calls a block with each one We can use that each operation without knowing its implementation from the outside hard coded collection and Generated collection both behave like a collection of the numbers from one to five We can define more collection operations on top of the one operation. We already have for example Here's a method called select that takes a block and then calls each and accumulates all the values that make the block return true So as long as a collection has a working implementation of each we can implement select on top of it It works the same for both implementations Of course in Ruby we have a module called innumerable that already has select and count and map and inject and Tons of other helpful stuff that all sits on top of this one each method So what do we mean by collection? Again, it's really just a specification an implementation of a collection provides an operation that follows a rule and Again, we get two benefits that each method gives us a common interface across many possible Implementations of collections and it allows us to build shared functionality like select that works with any implementation So what name do we give these things are they design patterns or interfaces or apis or duck types? All of those words are appropriate And they all sort of overlap to an extent and the concept of a stack or a collection kind of sits in the middle of all of them But in my opinion the most specific and therefore the most accurate term is abstract data type That literally means a kind of value which has certain operations that follow certain rules Stacks and collections are abstract concepts then but they're abstract in a good way The abstractions give us power and we expect programmers to understand them Nobody talks about stacks in hush tones. They're simple and they're useful Okay, so that's enough priming of your brain. Let's do some refactoring First I'd like to look at some code that has to deal with nils Imagine we have a project management app with different kinds of models Each project has a person who created it each person has an address each address has a country Each country has a capital city and each city has weather information which for the sake of simplicity Let's just assume is a string So let's say that in our user interface for this application We want to display the weather next to each project for some reason That involves traversing all of those associations So here's a method that does that maybe this is the kind of thing you'd write in a rails view helper There are lots of reasons not to write code like this But there are also good reasons to do it and anyway people will always write code like this no matter what we say If we make a city which has sunny weather and a country which has that city as its capital and an address in that country and a Person with that address and a project created by that person We can pass that project into weather for and it works fine But if we make a bad project for example a project with an address that has no country then weather for blows up So Tony Hoar invented nills in 1965 and he does now call it his billion dollar mistake Which he says has probably caused a billion dollars of pain and damage and this is exactly the sort of thing. He's talking about So they may be a mistake, but Ruby has nills so we're stuck with them To make our weather for method tolerate nills. We're gonna have to explicitly check for them So we need to introduce local variables to hold every intermediate result and then check each intermediate result Before we call a method on it While we're at it, we might as well include the possibility that the project itself is now Now this is turning into a bit of a pyramid of doom You can see the code kind of drifting over to the right, but luckily this code looks the same if we just flatten it So this code works, but it's pretty clumsy and it's hard to remember to do something like this every time We might possibly have to deal deal with nil Fortunately rails has got a solution to this problem So rails actually active support Monkey patches object and nil class with a method called try Which delegates to a public send if the objects not nil and just returns nil if it is So when every object in the system has a try method instead of doing all of these nil checks ourselves We can let try do it for us and now we're back to just chaining method calls together So we can take the local variables out again like that. So I'll just make that a bit bigger So this is as good as it gets right now It's better than the version with unless nil all over the place anyway, but can we do any better? Well Monkey patching definitely has its place But monkey patching every single object in the system isn't great. Is it this is kind of a code smell. Let's not do it So, okay, try's gone again now. We can all relax So when we want to add a method to an object so the good object oriented programming solution is to use Decoration and decoration is where you non-invasively add functionality to one object by wrapping it up inside another object So let's make a decorator class called optional whose instances have a single attribute called value Instances of this class just wrap up another value I can make a new one containing a value like the string hello And then I can take hello out again later if the value I put in happens to be nil I get nil out later Now instead of putting the try method on object, let's put it on optional If the value attribute is nil, it'll just return nil Otherwise, it'll send the appropriate message to the underlying object So now we can call try on the decorator and it will call the method on the underlying object as long as it's not nil If the value inside the optional is nil try will just return nil So instead of calling try on the actual project object and then on the actual person object and so on We can write the method like this We decorate project with an optional object and we call try on that Then we decorate the result which might be nil and call try on that And then we decorate the result of that call and call try on it and so on At the end we pull out the value and return it So that's unwieldy, but at least we're not monkey patching every object in the system anymore There's another smell here, which is that the try method does too much We actually just wanted to refactor away the nil check, but try also sends the value a message But what if we wanted to use the value in some other way when it's not nil? Try is kind of over-specialized. It's got too much responsibility So instead of hard coding the else clause here, let's allow the caller to supply a block that controls what happens next Now we can pass a block in to try and do whatever we want with the underlying value We can send it a message or we can use it as an argument in a method call or we can print it out or whatever and this ability to pass a block in to try is actually a little used feature of active supports try method as well So now instead of calling try with a message name and having to remember that it's going to send that message to the underlying object We call it with a block and inside the block we send the message ourselves and decorate the result in an optional And we could do anything else that we wanted with the value inside that block like print out a log message or whatever That works fine when there aren't any nils But unfortunately we've broken it when nils are involved because we're returning nil when the block doesn't run That's easy to fix instead of returning a raw nil here. We'll decorate it with an optional first and Now it works in both cases But there's a new smell Which is that I don't think try is a great name anymore because we've changed it to do something more general than or at least something Different from the main use case of its namesake inactive support So let's rename it to and then because it really just says start with this decorated value And then do some arbitrary thing with it as long as it's not nil So here's the new version of our method which calls and then instead of try and because we're just chaining and then Calls we can get rid of the local variables So this is verbose, but it's nice We decorate the possibly nil project in an optional object and then we safely traverse all of the Associations and then we pull the possibly nil value out again at the end So okay, how's our refactoring going? Well, we might not be monkey patching anything and it's conceptually clean But there's a huge final smell which is that nobody wants to write code like this In theory it might be better than active supports try method, but in practice it's worse But we can add some syntactic sugar to fix that Here's a definition of method missing for optional It uses and then to delegate any message to the underlying value whenever it's not nil So now we can replace all of this and then optional dot new with just normal message sends and let method missing take care of the details I'll just reformat that So there we go. This is actually really good You can see very clearly that we wrap up the possibly nil project into an optional and then we safely do our chain of methods And then we extract the possibly nil weather out of an optional at the end So to recap this is the whole thing an object which stores a value that might be nil and a method called and then which Encapsulates the nil check logic We added some sugar on top by writing method missing and if you're doing this in real code you should also remember to write respond to I'd like to very briefly point out that we only need to do the decorating and undecorating for compatibility with the rest of the system If the rest of the system passed in an optional object and expected us to return one We wouldn't even need to do that and then we won't have to remember to check for nil at all We could just write the method the way we did in the first place and it would just work imagine that All right, that refactoring was very detailed We're gonna do two others, but we'll have to skip the detail to save time Let's refactor some code that has to handle multiple results Imagine we have a content management application with different kinds of models. There are several blogs Every blog has many categories each category has many posts each post has many comments and for the sake of simplicity Let's assume the comments are just strings So let's say we want to fetch all the words from all the comments within certain blogs for some reason That involves traversing all of these associations again Here's a method that does that at each level we map over a collection and traverse the association for each object inside it When we reach each comment we split it on white space to get its words We have to use flat map here because we want a flattened array of words instead of a nested array If we make a couple of blogs which each have a couple of categories which contain some posts Which have some comments which contain some words and you can see here that my example accurately represents the usual level of discourse with blog comments Then the words in method can pull all of the words out. We're not worried about de-duplicating them or anything We just want all of the words But this method has got a bit of a pyramid of doom going on and plus It's hard to distinguish between the code doing actual work and the boilerplate of dealing with multiple values We can clean it up by introducing this class many which decorates a collection of values Like optional it has an and then method which takes a block But this time it calls the block for every value in the collection and flattens the results together So we can replace all the cores to flat map with instances of many and calls to and then and now we've got Instances being returned we can flatten the pyramid and Reformat the code a little bit to get this So again, this is pretty clear, but we can add some syntactic sugar by defining method missing This is exactly the same as optionals method missing except it's calling many dot new instead of optional dot new That lets us replace all of that and then many dot new calls with just simple message sends This is very nice We put the blog posts into a many object traverse all of the associations and then take the values out at the end And again if the rest of the system could deal with instances of many we could just expect one and return one To recap. Here's the class. We just made for our third quick refactoring. We're gonna tackle writing a synchronous code Does anyone near does anyone here know who the most influential Rubyist is? Now, well, let's find out once and for all We'll find out by using the github API to find the person who's made the most commits on the most popular ruby project When you make an HTTP request to the github API route you get back some Jason and it looks more or less like this Among other things this gives a URI template for finding out information about any organization So now we know what URL to use to get information about the Ruby organization When we make a get request to that URL we get some Jason that contains a URL We can use to get a list of all of the Ruby organization's repositories So we fetch the list of repositories which includes information about how many watches each one has from that We can see which repository has the most watches which turns out to be the main Ruby repository as you might expect and the URL for that repositories representation in the API and When we fetch that repositories information we get another URL that tells us where to get its list of contributors So then we load the list of contributors to the main Ruby repository which includes information about how many commits each contributor has made So we pick the one with the most commits a user called nobu and Finally fetch information about nobu from the URL in the contributor list So it turns out that nobu Yoshi Nakada has the most commits on the most popular ruby project. Thank you. Nobiyoshi Okay, that was a bit exhausting. So let's write some code to do it for us Assume we already have this get Jason method it Asynchronously makes an HTTP get request Passes the Jason response into a Ruby hash or array and then it calls a callback with the data If you like you can imagine a single threaded non-blocking event machine equivalent of this So to do what we just did we have to get the URI templates from the github API route Then fill in the template with the name of the Ruby organization Then get the organization data then find the URL for the list of its repositories Then get the list of its repositories then find the URL of the repository with the most watches Then get the information on that repository then find the URL for the list of its contributors Then get the list of its contributors then find the URL of the contributor with the most commits Then get the information on that user and then print out their real name and username name. So this code works, but it's drifting to the right again. It's hard to understand and maintain deeply nested code like this. But we can't flatten it because of the nested callbacks. So very briefly, the solution is to make an eventually class that decorates a block. The idea is that the block computes a value that might take a while to produce. And then the run method runs the block with a callback for it to call when the value becomes available. We don't have time to go into the details, but here's an and then method that we can use to add extra asynchronous processing to the value produced by an eventually. It's more complicated than the and then methods we've seen earlier, but it achieves the same thing. The main detail here is just making sure that all the callbacks get wired up correctly. So we can rewrite this code by putting each asynchronous get JSON call inside a block that we decorate with an eventually object. So we connect all of the eventuallys with and then, and then we run them. This isn't super readable either, but now we can start pulling out each logical part into its own method. The code that gets all the URL templates from GitHub can go into a method called get GitHub API URLs. This returns an eventually which decorates a block which will eventually call its callback with the result of fetching and parsing the JSON. So we can replace the line at the top with get GitHub API URLs dot and then. The next bit of code that fetches the data for the Ruby organization can go into a method called get org. This returns an eventually object as well. So we can replace the next bit of code with a call to get org and then the code that gets all of the Ruby organization repositories can go into a get repose method and then we can call it and so on for the rest of it. Let me just clean that up. So now that we're just creating eventually instances at each step, we don't need to call and then on each one immediately. We can let each eventually object be returned from its enclosing block before we call and then on it. So basically we can flatten this to get this. Let me just reformat that. So this is much nicer than what we had before. Each part is nicely encapsulated in its own method and the parts are connected together in a clean way. This might be a familiar pattern to some of you. It's similar to promises or futures or deferreds that you might have seen in JavaScript or event machine. So to recap, that's the whole eventually class. Okay. So what was the point of all that? You've now seen three decorator classes that all have this and then method. In each case, it takes a block and somehow calls it with information from the decorated object which could be nil or it could be multiple values or it could be a value that arrives asynchronously. All three of these things, optional, many and eventually are implementations of monads. And I think the most useful way to think of a monad is as an abstract data type like stack or collection. Like any abstract data type, it's got some standard operations. And then is the operation we've already seen. We need another operation which is a class method called from value. We haven't seen it yet, but it's very simple. It just takes a value and calls the constructor to make an instance of a monad in the right way. So this abstracts away the detail of exactly how to call the constructor with a simple value which is different for each monad. In addition to those operations, in order to qualify as a monad, there are some simple rules that those operations have to follow. The main rule is that and then must call a block with a value zero or more times at some point. And this is a much weaker guarantee than the one provided by each. It doesn't say whether the block will be called at all or how many times or when. Another rule is that the and then method must return an instance of the same monad and that's what makes and then calls chainable. The and then implementation in many and eventually already explicitly makes an instance of the same monad. Optional just trust the block to return one. We could have enforced that if we wanted. We could have put raise unless result is an optional or something. The third rule is and then and from value don't mess with the value. And that just means that when you construct a monad with from value, you're guaranteed to get the same value out again with and then. For optional, that only applies to non-nil values, but you should really only call from value with a non-nil value. The big benefit of monads is that they give us a common interface which allows us to do one thing, connect together a sequence of operations. The and then method means do the next thing. But the way it does the next thing depends on which monad you use. For optional, it only does the next thing if the value isn't nil. For many, it does the next thing many times one for each value. And for eventually, it only does the next thing once the value becomes available. And as you might imagine, there are plenty of other monads with different behaviors too. So like the size method for stacks or the select method for collections, we can define new functionality that sits on top of the common monad interface, and we can use that when dealing with any monad. So, for example, we can write a method called within. Within is like sugar on top of and then. Instead of expecting the block to return a monad, we expect it to return a single value, and then we automatically decorate that value again with from value so that you don't have to do it in the block. So because the within method hides the implementation specific business of putting a value back inside the monad, you can use it to write code that works with any monad. For example, here's a method that takes some monad containing raw JSON, representing a GitHub user, and then it uses the within method to parse that JSON and then assemble a string containing their real name and login, all within whatever the original monad is. If we feed it an optional JSON string, we get back an optional description as the result, and we know how to get a value out of that if we need to. If we pass in an optional containing nil, it doesn't blow up because the optional monad won't let description from even try to parse the JSON. It just immediately returns nil. If we make a many object containing multiple raw JSON strings, then description from will return many descriptions, and we can extract the actual strings if we want to. And finally, if we make an object that will eventually return some JSON by asynchronously fetching it from GitHub, then description from returns an eventual description, and we have to run that eventual description with a success call back to get the value out eventually. It's extremely useful to be able to write one method like this and have it work with any monad, and that's possible because of the common interface provided by and then and from value and the operations we can build on top of them. Like stacks and collections, monads are abstract concepts, but they're abstract in a good way. The abstractions give us power. As programmers, we should understand them. We shouldn't talk about monads in hush tones. They're simple and useful. By applying the power of monads wisely, we can untangle nested callbacks, make parts of our code more reusable, and generally make our programs better. If you want to play with the monad implementations from this talk, I've put them on GitHub at Tom Stewart. I've also packaged them up as a gem for some reason that isn't quite clear to me. If you're interested in seeing more computer science ideas explained with concrete Ruby code, I wrote a book that does that. I just found out this week that a Japanese translation is being published next week, so if you speak Japanese, that's all I wanted to say. Thanks very much.


What is up guys, so we are always looking for ways to improve our editing workflow, right? And there are literally hundreds of keyboard shortcuts in Final Cut Pro X, but which ones are the best? In today's video, I'm going to show you my top 10 keyboard shortcuts that I use most that helped me speed up my workflow. These shortcuts have proven me to be very effective time saving. If you are starting out, try to remember a few so that you don't get overwhelmed. Eventually over time, you can add more. I recommend writing it down. Try implementing those keyboard shortcuts as good as possible so that you can fluid edit your videos and use them in creative ways. So let's start off with the first keyboard shortcut. Adding a default title. You can select a default title of your liking for the shortcut control T. To do that, go to titles and search for the title you want to assign it to. I have the basic title set for this shortcut. I right click on it and select make default title. In almost all of my videos, I add a basic title. This is a really quick way to add titles to your videos instead of clicking on the title section and looking for the basic title. I went with the basic title because it allows me to create titles from scratch. It doesn't have to be the basic title. You can choose something different. I recommend selecting a title that you use most often. The second shortcut is adding a default lower third. So for this shortcut, the principles are the same. Again, for the shortcut shift control E, we can select a lower third from the title list. For this command, I picked the adjustment layer. If you don't know what an adjustment layer is, it's basically a layer that is put on top which effects can be placed into and will affect all the clips below it. I use this almost on every project, mostly to add a final look over the entire video. So for example, I finished editing the video, did the base correction and want to add an overall look to it. So I hit shift control E to add the adjustment layer. I then extended over the clip and add a custom LUT. After selecting the desired LUT, you will now notice that all of the clips beneath the adjustment layer have been affected. If you don't have the adjustment layer, I will leave a download link in the video description below. Our shortcut number three is delete left right part of the clip. I can just place a skimmer and press either alt five to delete the left part of the clip or alt six to delete the right part of the clip. Shortcut number four is in or decrease DB volume. I usually lower the volume when the subject starts talking and I have a music playing in the background. In general, you can select a portion by hitting the R key to select the range tool, then select the portion of the audio clip that you want to change. Hit control alt L and type in the values. For this example, I want to decrease the volume by minus 30 dB. So I type in minus 30 dB and press enter. This allows me to be very precise rather than using the mouse. Shortcut number five is duplicating. I use this mostly to duplicate my titles, but you can also use it for video and audio. Just select the video object and hold alt while dragging with the mouse upwards. Shortcut number six, zoom in, zoom out. You can use this to zoom in or out of the timeline. While we're at it, hitting the shift Z will zoom the content to fit the size of the timeline. Our next shortcut number seven is disabling a clip. Sometimes you want to turn a clip off, just select a clip, then press V to disable the clip. This technique is useful to quickly see how a scene plays with or without B-roll. Shortcut number eight is creating a shortcut for automatic speed. I have assigned automatic speed to command S. What this will do is set the speed of the select a clip to match the speed of the project. So if I have a clip that was shot in 50 frames per second and put it in a 24 timeline with the shortcut, I can stretch it out and play it in 24 frames per second, giving me that slow motion look. To create that command, go into Final Cut Pro, commands and customize. In the search bar, type in automatic speed, select it and press the shortcut you want to apply it to. Shortcut number nine, create a shortcut for audio fades. Same goes for the audio fades. I find this really useful. You can just select the audio clip and use the new set of shortcuts to apply it. I have set up three custom commands, which are control option X for applying audio fades on both ends, control option Z to toggle audio fade in and control option C to toggle audio fade out. For our number 10 shortcut is inserting gap clips. A useful shortcut for inserting a gap clip is pressing option W. I often use a gap clip at the beginning of a timeline to serve as a place to attach the opening music. A gap clip is also created when you push a clip in the primary timeline up to become a connected clip using the shortcut command option up arrow. You can also push a connected clip down to the primary storyline using command option down arrow. I use this a lot if I want to clean up my timeline. So these were my top useful keyboard shortcuts that I use the most when editing videos in Final Cut Pro X. I hope you've been able to find some of these shortcuts helpful. One other advice I have for you is to force yourself to do the shortcuts. If your mouse reaches for an often used item, look at the shortcut next to it. Go back and type in the shortcut. This way you can memorize it quicker and edit faster the next time you open another project. Thank you so much for watching. If you like this video then leave a thumbs up and subscribe to my channel to get the latest video. See ya!


large block sizes. I think that this picture represents well what Folius allows us to somehow get done. I don't know if you guys saw the movie Tetris recently? Anyone? Well yeah I think that 3D Tetris is one way to look at what Folius enables us to do at the block layer.  That's the goal at least. So large block size efforts really is an example of standing on the shoulders of the giants. Every time this comes up, every single LSFMM for what last 16 years I think. This has been you know tackled and the first patch set was by Christoph Lemender. Dave Chinner worked on this too. So it's certainly things that folks have been thinking about for years. There is a wiki and there's documentation there. I don't need to provide you the link right now. You can just Google it. I think it's the first entry in Google right now. Let me know if that's not right. But an example for just technology shifts here is the shift from 512 to 4k. Well you know times change right? We have different technology reasons for why we're considering embracing large blocks. But I'm not gonna get into that because we have a lot to talk about. So I'm sorry for that. XFS has supported large block sizes for many years but that's not the context under which Christoph Lemender first posted his patches for it. If you look at the mailing list and the title of his patches it clearly says large block sizes and then he goes on to explain how the goal is to support large block sizes as in this but when you're on lower page size. So yes this was supported for years and in fact my understanding is that there were some products that were sold on power for years on XFS. But you needed a 64k system. The goal of course is to support this on 4k. This is, I use Kdobs obviously for my development and I provide four NVMe drives by default so we skip that. If you enable an experimental parameter for Kdobs you will get additional blocked devices with each one more of a power to. This represents the LBA format that they support up to and the default that it's formatted. You can query the LBA formats that are supported this way and you can format an NVMe drive this way as well. Yes you can format the drive, yes you can boot but they won't work. And one of the things I was mentioned on the mailing list is that if you actually enable the large block devices for NVMe your system crashes. So we started looking into that and that's how you know we delved into this world and this is a status update of that, that effort and a lot of the dialogue that's happened here. There's a get-tree that I'm going to be moving forward trying to you know pick and cherry pick all these patches that put people post that might be related to help with large blocks. There's also support in Kdobs to do experiment with this as well including pure IO map which allows you to essentially boot into a world where you're just using IO map with Christos patches for instance and you will essentially not use buffer heads at all. You can test today large block sizes then using NVMe with large block NAX for instance. You can also use it with BRD and TempFS. So to answer your question that support is at least from our R&D perspective of course there's further you know R&D needs to go forward and you know further RFC and so forth but there's a lot to talk about so we need to move fast for this. Now let me show you guys where we're at. I like to try to keep track of stuff using OKRs.  I was actually going to address the folio conversion today not tomorrow. Tomorrow is actually the IO map stuff so this is a small list but this is actually a bit straightforward. I don't think this is really that complex so let's get into the meat of the stuff. The IO map conversion for those that are still struggling with that there's a session tomorrow to try to help document what the heck IO map is and how to convert file systems over. So if you go to Google and look for a large block size just replace large block size for IO map you'll see now so hopefully some sensible documentation or something coming close to sensible. And so I'll mention a few things here. Hannes's presentation kind of did address you know sun setting buffer heads but didn't address the block device page cache and it's important because as people pointed out it's used also for metadata IO. It's not clear if we'll ever get to a point where we're going to remove buffer heads. And that's fine because if you want to support large blocks of devices there's a way out and that's basically just to use IO map. That means that if you're a file system and you do want to support large block devices you likely do want to consider a solution for this. Otherwise then you know we will have file systems that do support a lot blocks of devices and we will use a pure IO map path. It's going to take a while to get there but there's I think a path. There's a bit of the efforts here I'll describe. If folks have interest in porting over stuff or let me know I just want to keep track of you know what how things are going. On the block layer Matthew's famous last words we don't need anything on the block layer anymore. There's quite a bit of work there as we have ended up discovering. So I think we have agreed that we're only going to support one order folio. I mean we're zero order folio on buffer heads so that's not going to require any effort I guess we can cross that out. IO map large blocks I support it's a community effort so it's essentially this will be moving forward you would want to test a pure IO map path right so you want to build a kernel without buffer heads. Yeah you can shoehorn in IO map as well to a block device but it's a bit hacky you would have to do that yourself. So let's see what else make buffer heads optionals that's Kristoff's patches right. The Ritesh posted patches to improve performance as well 16 times performance apparently that's pretty impressive so there are considerations there to use IO map on 4k for instance on 64k page sizes and then there's all these other things like for instance one of the things hallway tracks that I talked about with Kristoff was essentially the complexities in block cache and all these mix and matches. It seems that we probably should be moving the block size out of the super block and because we already have the inode with the block size so we should just use that. The other odd thing though is that you may end up in a situation where block device cache has a super block with a block size but different inodes have different block sizes that's kind of odd so to fix that we should probably just have a super block per block device instead of having shared one right now we have one shared super block and eventually the other thing too to try to help with read ahead and the page cache is to this was Willie's idea actually so this was to add the page order requirements on the address what is it the address address space I think one of the things that we should review is the implications of going down some of these paths and that is that essentially if we want to support large block devices we will have essentially an IO map path. All systems that do want to support large blocked devices will have to consider either solution for buffer heads or you know try to you know I don't know come up with the library option you know it's it's not clear to me you would have a better idea on that. Yeah so I think one of the things that wasn't clear to me and I actually jump back to the original topic proposal to try to get a sense for this is just to take a step back the topic proposal talked about how all the pressure is coming from the storage vendors and so I'm just trying to understand what's the business case rationale right you're asking for volunteers to do an awful lot of OKRs and normally you know what at least in corporate speak when we talk about OKRs we always have to talk about the business justification and I'm a little unclear on what are the use cases so where we would want a 32 K or 64 K you know physical device with that kind of physical or logical block size and we kind of skipped over that part. So I don't use OKRs for business reasons I use it for my own personal development I use it for everything that I'm maintaining cooling modules and stuff and allows me to go backtrack and not go crazy about things that I think I forget about so I don't do it for business reason this is more for community help and try to track things you know for the community I'm not asking for volunteers for people do things I think that people are already doing some of these things I'm trying to track what people are doing in consideration for a large blocks I support. Right but many of us work for companies and if I'm going to go to my company and ask them that you know some of the community who is working on company time should do some of these individual projects. So think of it this way. The justification because we need it for something else. What is the justification for supporting large block devices. This is a guy are we doing this. This is a dialogue if you want if you do have a reason to support large blocks devices. This is an outline of work that could be done. Could I just point out that somebody representing CloudVan does yesterday said that a 16 K block size would really help some of their workloads. Yeah so what we said is that we need to support 16 K database page sizes so we have outlined something that we can do in reasonable time that does not require us to use large block devices. So because like there's a cost benefit profit justification for it and I can't think of a reason why I should spend corporate time working on this. There's a simple answer to it. Then don't. Yeah so this is yes experimental. Yeah and this is something that we think might be good idea which we think might be fast or even better in whatever for whatever reasoning for better. But this is something we simply wouldn't know until we tried. So all right. Let me make it a bit clear. If you do only want 16 K high order or aligned block device support for even though your device is 4 K guess what you're probably still going to end up getting a large block device just that it's painted in a different you know color. It's the same thing. We don't need to do all of the infrastructure. No no no just outlines. Yes. Yes. Yes. R's right. That's this I see like huge headcount requirements for this and I'm trying to figure out how we justify the headcount. I think we're getting off the rails here. So like let's let's talk about what we're doing. Yeah. We're talking about business stuff. We could talk about that on the side. This is not about business. This is if you do want large blocks I support. This is likely what you want to look into. Ancient file systems. I mean this is not just a large block device support type of thing. Right. You know there's already dialogue on removing Riser FS and I'm not sure if that's ever going to happen but it seems like what's four years or something like that that we're saying to remove file system. There's dialogue also possibly moving old file systems to fuse. There's complexities also in conversions of older file systems as well. So for instance I was told that there are some file systems that we don't have MKFS utilities for them. So testing that seems really complex right because you can't really recreate those file systems unless you just get the image dump somehow. It's really limited. Not even sure why we ended up trying to support those file systems but anyway if you have ideas on old file systems and progress on that go ahead and look at the page cache. This was the stuff that I just mentioned earlier regarding the inode address space using block bits and there's a link to Willie's comment on where that came from. And then we have the higher order folio support rationale is again on the wiki just look at that. Memory compaction came up but in trying to talk to Blasphemy it's not clear to me really if there's anything that needs to be done there. Willie? As I remember the memory compaction code has not yet been converted from pages to folios. I think it's more the allocation of fresh pages so right now I don't think it does very well at migrating pages from migrating non-zero order pages from one zone to another. I think that's just something that needs to get fixed. I haven't looked into that in any detail but I do think it is a problem area because it's still working in terms of struck page. Once it's working in terms of struck folio I will have delve deep into it or somebody else did because I don't have to be the one who does that work. Okay one of the things that became evident to me at least when doing experimentation with Shemem or Tempefest with higher order folio support was eventually also addressing swap with higher order folios. That's the swap cluster read ahead for instance and friends. There was a work also to evaluate Chinner's effort to support larger block sizes when it's greater than the page size and I essentially just tried to rebase all that work and Chinner basically ended up providing good feedback there and it seems there was only two patches really needed there. There's testing ongoing. First I want to test that to ensure nothing breaks. Make sure that there's a baseline there for at least XFS and a 4KN drive with larger block sizes and if there's no issues there and then hopefully try to see if we can promote getting that upstream once that's present upstream then maybe we can evaluate testing XFS with large block size on a real block large block device. BRD, Hanes has patches posted seems like he'll be following up on that and there's I'm not sure if other file systems are interested in supporting larger blocks sizes are there? So weirdly enough we already do this by default for metadata only because data is such a problem but metadata we default to 16K block size basically and so we just then you can go to 32, 64 or whatever but like because cow is unfortunate sometimes it's better for us to like do this in big 16K chunks so like you know your default Fedora install has 16K metadata block sizes on 4K page size. We'd love to do this for data as well it's just data is trickier right now. Right right. Okay so it's not it's not on the roadmap right now but it's metadata certainly. Yeah metadata has been there for years it's been working well. Data is we're doing IOMAP first and then hopefully IOMAP gives us everything we need to do. I had not considered metadata requirements for file systems that would be interesting right because right now we do the control for high order folios on the block size right and the question would be how to do that for file systems for metadata. Yeah this is where that like you know metadata abstraction thing that we have we just have like this like an array of page pointers that we we do so like at the point that we can start allocating full 16K folios and just drop that in there instead of having like 4K things that would be cool. Is the address base somehow used for metadata or not at all? That is horrible what we do because we have a lot of metadata we're not like EXT4 or XFS we have gigs of metadata like you know hundreds of megs of metadata that we write out at any given time so we have a fake inode with an address space that we hang everything off of and then we just set the aops to like nothing and then we manage it all internally. If you remember a few probably five or six years ago I like wrote a bunch of code to do like byte size throttling so balanced dirty pages this is the big problem that and this is why we do it this way is because with as much dirty metadata that we generate we can overwhelm the system and you know use too much memory and so we need to rely on the balanced dirty pages we didn't want to like redo all that work so we use the inode to take advantage of balanced dirty pages and I tried to do this in a generic way and I just got distracted by higher priority things but that's why we do it. Got it thanks. Yes. So to come back to Ted's business case thing which is basically this looks like a lot of work your chin a thing you said there are only two patches I remember that patch said it was pretty huge yes is what you're saying that once we've done the folio conversion actually what looks like 15 million man years of work will actually fall out that's correct that's correct essentially essentially your file system should be a lot easier to convert and add support for large block size support if you want that one of the issues of course is the metadata stuff right and currently using bufferhead yeah I think if Derek is on he can talk about what work might be needed for XFS and okay because I know you talked about this last week about supporting XFS block size larger than page size and maybe you could comment on the XFS metadata buffers or the XFS buffs that would be so as was previously touched on XFS has its own buffer cache hidden inside of its code base so we don't use address spaces or any of the strange things that butterFS does with address spaces so as far as metadata goes we already support having file system block size greater than page size it works at least until memory fragmentation kills you the only part that we that actually doesn't work right now is the part is the IOMAP part because we don't have a good way to tell the memory manager hey we want both multi-page bullios and they have to be at least this size because right now all the IOMAP code kind of assumes that the block size is always less than the page size which we can keep true if we could have if we could require multi-page bullios of say you know 8k for our 8k file system block size so I think what Luis is talking about when he says that this huge grody patch set collapses to two is that I think all we really need to I think the only piece that's left is just making the the memory manager give us 8k say pages or 8k folios and then seeing what falls out of the system once memory augmentation comes up and tries to eat us alive now you know Matthew has a has a theory that if everybody uses 8k in the system then actually it will be fine and we'll just reclaim things as we do now I suspect there probably are going to be other weird issues involving making it clear to the memory manager that if you want to reclaim part of this 8k folio say you have to reclaim all of it not just one of the two pages you can't really basically you can't split large folios down to something anything smaller than whatever granularity we established in the first place but the last time we actually anyone actually tried running XFS with 8k blocks on x86 obviously with the file data paths disabled it worked fine other than you know toy XFS became the toy file system where you can create directories and extended attributes all day but you can't actually store it read or write anything to a file so I think as far as XFS goes we're nearly there we just need some things that have traditionally raised eyebrows amongst the M.M. folks about hey are you got what kind of crack are you guys smoking hey I just want I just wanted to touch on something you said there when when the M.M. is is reclaiming memory it's always going to take an entire folio so it's always going to reclaim an entire folio so as long as the the the tricky part is keeping the M.M. from fragmenting larger chunks of free space so that's that's that's that's where we need to do and Johannes and Mel are currently arguing about a patch series on Linux M.M. that will try to do somewhat better in terms of keeping large chunks of memory available. Mm-hmm yeah I mean I think I think the answer is we really just have to stand up a test a bunch of test code to actually see what happens by you know make make the theoretically small changes to XFS have somebody stand up my sql or something on the on a system with the ridiculously small amount of memory and see if it see if the whole thing falls over any sooner than it does with 4k pages and 4k file system blocks. Right it gets it gets to the question of how to how to determine that easily you know what sort of testing do we want to do how do we detect that how do we measure fragmentation a lot stuff how do we replicate these sorts of you know dire situations that everyone is afraid of how do we replicate that how do we exacerbate that you know what do we do what's our test plan we got to really sit down and just carve that carve that out and yeah that's a part of the next thing here is testing basically we want to get to the point of testing first XFS on a baseline on 4k and drives with larger block sizes right without 4k and drives but then what does it mean when you start enabling it on a real large block device test how do we stress that how do we measure fragmentation how do we cover these things what metrics do we have available you know and this is part of a community dialogue and this is why we're here so maybe we can talk about that for those who are interested in letting talks on the side and so forth but that's what I have so questions hey Lewis this is Hitesh am I audible yeah yes so I do have you know a point to discuss look so is this the similar problem that I mean like should we have something like a multi-order block size like for example the similar problem that you know the memory management is is facing with multi-order folio so so so what my point is like for for data blocks like for example in exe4 we have a big block where where we can actually do say 16k blocks to be written but for metadata you know can we still can we still have 4k like rather than supporting you know I haven't looked into the 16k you know stuff from exe4 so I was referred to look at the youtube video so I will have to go do that so tell me be able to speak about that yeah basically like my point was that should we consider you know having different block sizes for different data types like for example if you have data you can have you can actually go and write a 16k whereas if you have you know metadata it is expected to basically track track that at 4k yeah so the reason why exe4 did big alex and we did this many many years ago was because the metadata problem was deemed too hard tm and the mm fragmentation problem was considered too hard tm and so big alex just simply allocated on disk chunks uh you know chunks 16k chunks but the assumption was the page cache could still be 4k and it would be fine in a folio world it'd be really easy to have a 16 or 32k big alex exe4 file system uh if we had a way to tell the folio system by the way this is a you know big alex 16k file system all the folios should be 16k or larger um it'd be really easy to do that and we wouldn't change anything else because the fundamental metadata block size is still 4k and that would stay the same so if what we're used in a folio world um this is actually pretty easy um the way exe4 did big alex and that was simply because we decided to solve the easiest parts of the problem first we were not trying to solve the general case um so yeah if my understanding is correct and then perhaps the same solution that would be used by tempfs might be leveraged by exe4 which is to eventually use the inode address space for higher order folios um but be a bit hacky but you know it's a different way to look at it no no then i'm not sure any other questions thanks all right thanks