Hello everyone. Welcome back. It's time for us to do more proc macros because I ran a poll and everyone wanted more proc macros So that's what we'll do. I also ran a poll about whether people wanted to see chat in the video and The consensus was that we should not have it Keep in mind that if you are watching this sort of after the fact on YouTube There is a link in the description description below that links to the live version of the video Which will have the chat on that side that side while you watch And so that will be kind of neat So before we get started on today's stream I also want to point out that I tweeted this a few days ago that I Used to have a patreon for this work that I do and realize that I'm not allowed to have that in the US Because I'm not allowed to have any source of income outside my studies And that was a bummer so I've been trying to find a way because a bunch of people have been asking me how they can sort of support my work and And so I've decided to set up an Amazon wish list Where people can like if if you want to like donate something to help make the stream better you can do so And I posted this a few days ago and I've already been just overwhelmed with the responses it's been really cool to see people just like Strangers on the internet sharing things with me and so to all of the people who have sent me things. Thank you so much I appreciate it a lot it it makes me Yeah, it's great. It's very much appreciated And so there's a list where there's some small things and some large things is a bunch of D&D stuff because I like D&D And so if you want to buy buy me stuff from this list then That would make me very happy if you can't that's okay, too This is just the only way I could find where there's some way of showing support But even just like your thank yous mean so much to me I Also want to remind you about the live coding voting site So I'll post this link in the chat somehow In fact, how will I even do that I'll do that this way So On the site you can vote for upcoming stream ideas It my guess is this will be the last proc macro stream And so the next stream unless things change is going to be a second open source contribution stream We did one of these a while ago and the idea here is basically I Spend like the five hours or so that the stream goes for Trying to contribute to a number of different open source rush projects And I'll sort of solicit opinions from you the viewers as to which projects we should contribute to And it can be small contributions large contributions Whatever we feel like we want to do usually what we'll do is try to find less developed Projects and then look at issue trackers for example and then try to find a thing to contribute Let's see before we start Okay, so there was a question here Do you have any videos implementing event loop? So some of the future stuff is event loop based you could try that I Couldn't have a patreon because as I'm on a student visa in the US And so I'm not legally allowed to have any source of income that is not basically my The assistantship that my university pays me not allowed to have any other source of income Okay, so let's drive dive into procedural macros again I'll try to get better at like reading chat allowed as well So I'll be monitoring that in the background. So if you remember from last time we started working on David Tolna's procedural macro workshop from the Rust Latam conference And we basically did the very first exercise of deriving a builder pattern So the way this looks is you want to have some kind of struct and you want to be able to put derive builder above it and What our macro does is basically generate the code to have a builder for that thing So if you remember back to let's see if I can find this Just to sort of bring us back to where we were Oh 9 No, let's do oh Builder oh builder oh seven great So the idea is that if you have some code that looks a little bit like this So you have a struct with derived builder and you have some fields with different types. You might even have Things that are vectors for example where you want to build or to have a Builder argument that just gives you a singular thing Then that should give you a builder type that you can then use this stuff on and so to remember The procedural macros basically take as input what I do The procedural macros basically take this program as an input sort of a stream over the tokens of this program as an input the things that have been annotated with derived builder and then produces an a Stream that adds new tokens to the stream. So what that means is it basically expands the program So in this case cargo expand shows us the expanded version so here we have the We have the command that was in the original source Remember the builder the derived builder Attribute has now been removed because it's already been parsed and what we generate is a new struct command builder with the same fields a Bunch of methods on that builder and a build method that produces the final command and so what this means is ultimately this generates all the code that's necessary to make the the Operations that the user gave us work out So the question is If we go back to our list here The question is what what exercise do we want to do next? My hope is that we might be able to do maybe two in the five hours. We have Given that the first one basically taught us a lot of the concepts that we're going to need in order to do these and then the other ones are Sort of expansions upon those topics. So let's do a quick straw poll in chat here Let's let me first briefly go through the options. So there's derived custom debug So the idea here is that you know how you can have derived debug on a struct Custom debug is going to also let you define how you want different fields to be debug printed So in this case, this would say print the bit mask using this format So that would be one macro we can implement another is seek so the idea of this macro is that you can give the seek macro followed by basically an iterator and Then a segment of Rust code that will be Where we're going to look for this pattern This like pound open brackets similar to like in macro rules and what's inside there will be expanded n times where n is the iterator with pound n replaced with the value of the iterator Right, so this would generate like CPU 0 CPU 1 CPU 2 CPU 3 etc all of them as variants of processor But it doesn't have to be inside of an enum. It could be all sorts of different things pound sorted is Going to be a compile time assertion that the variants of an enum are ordered And so if you put pound sorted on an enum, then if the enum variants are not sorted, it will be a compile time error Bitfield is Is I see you basically define how many bits each field takes up and then the the attribute Let's you access those fields and set those fields appropriately and make sure to store them using whatever the appropriate integer type is And my guess is that the bit fields one is probably a decent amount more complicated so Looking at the project recommendations here from David Custom debug is going to look at trade bounds seek is for Custom syntax, right? So normally we've been using the the syn crate for parsing out rust syntax in the case of seek We're gonna have to basically walk the tree the syntax tree to find places where we need to replace the main variable Right, so this is basically like custom in tax parsing Sorted is mainly for looking at how to get good diagnostics from a macro and Bitfield yeah, so bitfield do only when you have a strong grasp of at least two other projects So I think we should We might do bitfield today, but we should not do it first And so I think the options are do we want to do sorted? Do we want to do seek or do we want to do derive custom debug? All right, so votes who thinks we should do which Specifically out of these three and not bit fields we might do bit fields after but I think we should do at least one more to like Get back into the group of things Sort of a seek seek seek is probably rather pronounced like sack than seek sec Maybe so the reason I pronounce it seek is because it's sequence Right seek sequence It's not sequence right or a sequence, but I don't know Sorted looks nice and easy custom debug sorted seek Okay, it looks like there's sort of a tie between sorted and seek maybe I Think seek looks like the Contender here. Okay, so let's go for seek first and then we can always do the others later I agree that sorted is probably Easier right because it it own it's only like enum variants and checking with the sorted All right, so let's get started with seek function like macro So the idea here is we're gonna be given Sequence We're gonna define a function like macro seek, right and it's gonna have the format Like this it's gonna have to require numbers Stamping out sequentially indexed sequentially indexed copies of an arbitrary chunk of code All right so Great, so we're gonna have to parse basically this followed by An Identifier the keyword in a numeric range Curly brackets and then just regular code and then inside of that code segment. We're gonna have to look for this So let's see how we might do that so the syn crate if I remember from last time has a visit Thing which is basically a way to walk a syntax tree and in this case we want to Mutate the syntax tree, right? Actually unclear. I think that's what we want to do Because we want to replace we want to replace the occurrence of this with the expansion of it, right? So visit mute So I think the idea here is Visit mute This is not at all documented That's potentially unhelpful Wonder how we're supposed to use this so does it say visit anywhere else here Not really Actually, maybe visit is documented and visit mute does not let's take a look at visit that seems relatively unhelpful Is there a visit token stream? Maybe like a top-level entry point? Are there provided methods for this where? Oh, yeah It's a good point all of these Things from the workshop are actually there are real implementations available of them. I don't know where they're linked Think if you read this more carefully than I did basically all of these exercises There's a crate on crates.io by David that implements this feature And the real question is where is like the top-level entry point? Where do we start? Is there there's no like Is there a token tree? No Interesting. Oh, maybe there's a parse a variant of parse that takes visit parsing a custom syntax Lacy static example Great Take a look at that. Well, we don't want since like that's Actually, this might be useful for parsing the This might be useful for parsing the top-level thing right so For parsing this business right that business is sort of like lazy static in that we'll want to parse a particular sequence of tokens Okay, so that works decently well Mm-hmm Getting error messages. That's fine the real question though is Ah, this still doesn't really give us the visit pattern I'm like fairly convinced we want the visit pattern, but the question is how do we start parsing that way? Actually, I wonder whether this would let us parse module Maybe something in here takes visit parser trade No, this all seems to just be to turn token streams into parse streams um Wonder is what we'll do for positive voice a fold visit visit mute Is there a fold in there? So for I guess visit mute Do you see a fold? I did not see a fold This is a top-level fold To transform the nodes of an owned syntax tree. Oh Interesting is The fold might even be better Yeah, you're right. I think fold might be a Might be better because that's that really is what we want to do we want to walk the input Transform it and produce an output But it sort of has the same problem in that these aren't really documented like none of these give me an example Of what they do although the the function type sort of helps Fold macro Specifically the question is What is like the top-level function I call to fold over a stream? So what I wanted to do is look at this and see how this does it So here, okay bunch of documentation. That's fine Right, so this is parsing that This is basically the same code we're gonna end up writing Expand repetitions is really what we want to see Yeah, so the body here is the token stream, right? The token stream is sort of what we care about and Interesting so this doesn't actually use the visitor pattern which is surprising to me Instead it just walks the groups so if it's a group then it does that Huh interesting, I guess what we're really gonna need here is Proc macro 2 So remember proc macro 2 is the crate that gives us Accidents of the procedural macros in the compiler in like a way that's independent of the current compiler version And the token stream Basically gives us an iterator. Where's the here an iterator over token trees that are either groups items punks or literals Okay, well in that case. I'm very surprised. We can't use fold here I feel like we probably can the biggest problem is that fold isn't really documented. I Wonder whether the trace var example uses fold. Oh nice. Okay, let's try that then Trace var Trace source and let's see here. Oh It's like expression the top level thing Item fn The I think the real question is what do we parse it as first macro into args as R? Okay, so that's fine. This is to parse If we go back here, okay, so this this macro Is an attribute always on a function and so that's a little different because Okay, so it parses the input like the body that it's given as an item Is specifically a function item it parses the arguments of the macro as args So that's reasonable and then just fold item fn. The problem we have in our case Is really that for if we go back here We don't really know what type is gonna be here it can be anything right it's like any token stream Which is why we don't actually know where to start the fold Maybe we know it's a block Or we know it's an item. I Wonder if there's a general like fold item Yeah, fold item That might work an item is an extra and create a use a static a constant fn mod existential structs no because it doesn't have to be an item either Like our problem here is this could be anything right like the the stuff that follows see could be Just a sequence of tuples that he gives an example of here right it could be an expression It could be it could really be anything and it looks like fold doesn't have an entry for anything Like I what I really want is like a fold But I don't think there's a just a fold So it seems like we might actually have to just walk the The token stream ourselves It's a little unfortunate, but we can do it Okay, so we're gonna start with where's our no longer in builder. We're gonna be in Seek, we're gonna be in Here this is still gonna be cargo expand and Here Test Great What I want is just for us to have something to start with for our Our main so that we which we can use cargo expand with Right, so if I go here now run Actually, let's make this be So cargo expand Currently just crashes because we have ended up with the macro, but now at least we'll be able to There's something you want is called the token tree. Yeah, I know that it's a token tree, but there is no fold token tree Okay, so what we're gonna do is we have to start implementing this thing in the first place So seek source lib, right? So now we're at the point where we know we define a proc macro, right? It's not a proc macro derived like we wrote last time this time. It's a proc macro which defines a Sort of a similar thing to what you get with macro rules like a macro you can call And so that's what the proc macro attribute here Tells the compiler is that what we're defining is a function like proc macro And what that is given is it's given an input token stream and it has to return a token stream It's gonna replace the current item. So if you remember from the last stream a proc macro derive Appends to the token stream. It's not allowed to to modify. Well, I Don't think it's allowed to modify the input, but it produces a new input like appended input So won't seek zero one million be horrible for compile times Well, so it's true that if you were to repeat a particular thing a million times you're basically Expanding the source of your program a million times. So yes, it would be terrible, but then again That's what the user asked you to do, right? Okay Okay, so we know from last time that what we want to do here is parse the input to the macro, right? So if you remember In sin, there's this handy-dandy macro called parse macro input Right, so if we hear you sin parse macro input, then here what we want to do is parse macro input of Remember how the syntax is the identifier as the type Right in our case, we're taking the input and we want to parse it as So remember last time we used derive input, right? This time, that's not actually what we want this time we want Where's the oh, maybe the default is just what we want So what does it say here as my macro input? Interesting actually that is what we want to do but this is exactly what we want to do, right? We want to parse it it The input to our macro is going to be some custom syntax and so sure we're gonna do this Although it's not quite we're gonna give it a better name than that But first of all, we're gonna have to just like make it not crash, right? So here we're gonna return a token stream So the macro is gonna expand to nothing currently we're gonna parse it print it and then expand it to nothing And in our case, I guess here we're gonna have to do this business So this is gonna be a seek macro input Seek macro input, we're gonna parse the input we get as a seek macro input and here this is gonna return Okay Let's see what that does. Oh, do I actually need to This All right, I haven't even defined these dependencies I want oops. Nope. I want dependencies to be seek 015 and also Quote 06. Oh I meant sin and here we want to derive Debug great Unexpected token, huh? Oh, I see because now we're not actually consuming the input And so it thinks that the macro basically starts parsing here And then it consumes no tokens right because our parser does nothing And so it then says okay, I'm done and then the compiler encounters this token is like I did not expect there to be a token here The macro said it was done parsing, but it doesn't moved along in the input stream So what we need to do here is basically the the same thing we thought we saw earlier, right? We want to basically parse the custom syntax that we're expecting So if we go back to here, we basically need to parse out this business, right? And so what that's going to use is if we look at the parse module It basically provides a number of parse functions I don't actually need look ahead this Um, was that there's a nicer Yeah, so you basically give Oh, I see how they've done it. That's interesting so There's parser and parse. How do they different? Oh, we might need the parser module. It might not be on by default Yeah, so parse Is anything that can be parsed in a default way from a token stream, right? So we're going to have to say something like if we expect the next thing to be a literal, for example Then if I understand this correctly, it's going to be something like the variable we want Is going to be Actually, no, it's not going to be token. I guess the first thing we're expecting is an identifier for the the variable name the n, right? So it's going to be syn ident parse input Let's see what that var is Hopefully this should print n Let's see what it gives. Yeah, so this gives us an ident n and it shows the span Remember from last time which tells it where in the original source that ident appeared Right All right, parser. You're right parses will be implemented But I was wondering what parser is because all of these use parse, but I don't think it's important Right. So if you look at the index we're syntax we're expecting we want n we want the The literal token in then we want these things So next question becomes how do we get a keyword like in From memory there's a token Thing, uh, let's see if I can find this somewhere Is it macro? Token Yeah, so I think this is what we want. So here let's bring in token as well So in is going to be token in parse input And I guess we can we can print that out too, although this one is less important, right? We don't actually care about the value of in we just care that there isn't in there Uh, sorry, I meant Oh, that might not work. Yeah, although that's not actually what we want either interesting Because uh token in Gives me a I'm guessing like sin token like it is basically a rust keyword, right? It gives me a concrete one. So I don't know how to call parse on that Oh, it's probably something like this Is uh input dot parse Not sure Oh, is that what you wrote? Oh nice great Right because token in gives me a type although that should then still work actually If it gives me a type then This should be the same But the compiler might not be able to figure out Yeah But if we do this, I think that should be the same Basically telling it this is a type called this associated function Great, that's the same perfect Uh, so the in if you look at it just parse to in right Just a keyword. We don't actually care about the value Which is why I use the underscore so that the compiler won't warn us about unused variable All right, so the next thing we're expecting is We want a literal followed by two dots followed by 512. Okay, so we're going to have I guess From it's going to be a literal parse Two is going to be a literal parse And in between we're going to have dots double dots Oh, and I guess I want to print out in fact, let's just keep all of these in one Um So var in from dots to We miss any I think that's right Right, so we get the ident n right we get the in keyword We get a literal number zero Then we got Dot two right so dot two that is a double dot which is a literal Then we got the literal number eight And that's what we parsed and then it's saying unexpected token for the the curly brackets And so now what we want is the next thing that's going to come is basically a block Right, so the question is how do we parse a block? What is the type for a block? Which let's look at our parse here Actually, this might tell us the type we're going to use for fold Whatever we parse here. Actually, the body is just going to be a token stream Right, the next thing now is just a token stream of some kind But is there a way for me to say like curly brackets? Um use tree is not what we want Um Expert block not really necessarily an expert block I wonder whether it'll let me It's really almost like a module block Like an item Item block is that a thing item mod maybe No, because that needs to include the name. So what's the content? It's a tuple of brace And I and a vector of items Oh interesting Uh, you could probably also parse as an expert range So the reason I don't want to parse it as an expert range is because We're going to have to iterate over this range Uh, and I probably don't want to allow I don't necessarily want to be able to support any Range although you're right. I mean we could just assert that it's a particular type of one Um, so what is expert range parse? Okay, any of these? Yeah, but like, okay, so imagine that I got this back, right? That's not a useful construct for seek. It can't generate an infinite number of things. We don't really have iterators here. We're gonna Copy the same thing multiple times. The one thing that might be nice is that we get this But I don't think it's terribly important There's an input parse for block Yeah, but it I think a block is like an expert block I mean, what what is a block? Abraced block containing rust statements. No, you're right. That looks perfect. I agree with you Okay, so the next thing we're going to have is body which is going to be soon block this But that is not an input to fold Which is weird What? Oh, I need the full feature Okay, fine. So Version is this Features is And then you can continue with sub input Wouldn't block have to be completely valid syntax though Oh, that's a good point Block might actually part. No, I think would That's a good question No, I think it just parses it as syntax and like remember a macro invocation is also valid syntax, right I guess actually we're going to find out. Okay, so we now pass my guess is we pass test one Oh, what was the r s progress Great. So we're now going to pass the first test. We should pass without a problem because basically because we don't crash You need sin braced Oh, do I need to explicitly enumerate all of them? What are the default features? I see. So I probably what I want here is just like all features, but let's do full Why does it work in Cargo expand but not in cargo test Oh extra traits, but why But why does it extra traits I wonder why it works in cargo expand. That's weird. Oh cargo expand probably also relies on sin with that feature added is why So what cargo does when it builds something it it if it finds one dependencies and multiple parts of the dependency graph That have the same version it takes the union of the features for it And so if cargo expand builds with a dependency on sin with extra traits turned on Which it might very well do then it will also compile the underlying crate with that feature on which is why this compiled fine and also why this this Was not working great. So we now pass test one. So let's go to test two I guess progress Let's do two Okay, so this what does this expand to Um, okay, so I'm guessing this won't actually pass Wait, what? Oh interesting The macro invocation of the previous test case can be an empty loop body inside the brace in reality We want for the macro to accept arbitrary tokens inside the braces Colors should be free to write whatever they want inside the braces seek macro won't care Whether they write a statement or a function or a struct or whatever else So we'll work with the loop body as a token stream rather than a syntax tree. Yeah So this is what was pointed out elsewhere that um That what we might want is to not even parse it as a block But just as a brace like just It is contained in braces, but we don't actually want to parse the thing inside yet Although that still means we can't do it with fold, which is a little sad So if I go to lib here Um So instead of this what's the if we go to here Braced Interesting What is that what does this do? Why does it need this in syntax? It's interesting Brace token is a type token brace So why does it need Braced That's super weird Oh, I see it just consumes the braces, but why is the let content necessary? like specifically um Specifically is it fine for us to do like let braces is in uh Where's token Like token brace parse input Like is that a thing That it's going to be okay with Right, I'm guessing why doesn't it just expand to its own value? It's the real content Uh, I kind of found Body what do I do? Oh this business Okay, so brace can't be parsed. So it's really that's so weird. I don't understand why Why it has to be cold this way braces input as content content as input Like why is that necessary? I guess this is braces Oh, the content is the stuff that's inside Never mind, but I I still don't see why it can't be this right. This is what I really want um Oh because you also need the braces. Okay. Oh man, that's that's funky is what that is Okay, so this returns the braces and I have to write this to say I want the content to be placed in that variable Oh, that's awful But I see why it's needed Specifically the problem here is braces really needs to do two things It needs to give you a handle to the content inside the braces and it needs to give you a handle to the braces themselves So you can replicate them if you need to This means that it needs to return a value, right? So one of those can be returned, but both can't be well I could return a tuple. I guess Actually, wouldn't that be nicer? Wouldn't it be nicer if this just returned a tuple of braces and content? Oh in right um Oh, well, okay. So this means that the Content is the stuff that's going to be inside the braces uh Braised Question mark operator cannot be used here Wait, really? That's fascinating. What if there aren't braces there? Why doesn't this return an error? um Right, okay, so now we're not currently parsing the the Token stream inside I guess how do I can I just say that I want to consume a token stream? uh proc macro 2 If I have a token stream I guess I could just walk the entire thing Always he braced contains a return statement. That also seems weird but okay Right, so the real question is what is the stuff that is inside of? Inside of the content so inside the braces well, I think what we want here is um Right, so the content expands to a token stream. It's going to have whatever kind of syntax in it Um, so here what I think I want us to do is just for in content Um Hmm Oh, right. That's what braces give us. So if you look at um It gives us a parsed stream Not a token tree So if I go here go to parser This is where parsed stream is parse buffer Right a cursor position within a buffer token stream Yeah, I so okay the Content is now input. I agree. I I understand the the setup that the Point now is content. So the the error we're now getting is In parsing We're currently looking at over here right, this is the current file we're trying to work on and currently what we've done is we've parsed everything up to the curly This curly bracket and this inside here is still a token stream that we're not consuming anything of And this is really what content points to what I've just highlighted That's what the content variable is the a parsed stream over that content Um, but currently we're not consuming any of it. So our parsing basically stops here And when our parsing stops there the compiler then says Up here unexpected token expense to nothing right because we haven't told it that it should expect to parse that out Um, and so one thing we could do here, of course is we could just do like while Is there like an is empty? Yeah, okay. So while not content is empty And then like Yeah So will this even I don't know what this will give us I guess let's see Yeah, the problem is we have to actually say what we want to parse it as right Oh, someone suggested the same thing nice um Yeah, Felix. I think no, no, it's it's super helpful what you do is more like I end up thinking through the same thing as you do while you type it Although it is useful to have someone else who knows this during the stream So let's see the the real annoyance here is that we don't know what to parse this as We sort of really just want it as a token tree um Actually, let's take a look. What did the over here? Yeah Because in david's version he keeps it as a token stream token stream, which seems like the sort of right thing to do That's sort of what we want to um Oh, that's funny. Yes. He parses out a potential equals there Um require braces So this seems to be much lower level. Does he even use sin for this? Oh, he just parses it manually. He doesn't even use the sin crate for this So that's part of the reason why this gets complicated um Whereas we're trying to do use a do it with the sin crate So what does he require braces? Where does require braces come from? Create parse parse require braces um Yeah, this is why we might not want to use the um That's why we might not want to use um The sin parser for this because here um What we really want to do is just get the raw contents within the Within the braces Unless unless we can go from a parse stream to a token stream Right like uh for parse where's our where do I have parse open here? Oh, yeah So what we can do here is actually just um Something like let token tree is going to be Rock macro Token tree Parse input uh content Let's see what it does with that Uh, which token tree is this? Oh proc macro 2 right now. I need to also depend on Proc macro 2 which is what version proc macro 2 is version 0.4 Expected reference Really apparently Uh, but it can't be a token tree either token stream right because a token tree is a single it's Not quite a single token, but it's like um It's basically one subtree of the AST So if you look at the error we got was unexpected token was now exclamation mark And that's because this is an ident. So this is an AST in and of itself This is a separate part of the It's the basically the next token Right this would be one token to be a group that's delimited by brackets And so if we park it as a token tree all we're really parsing is this If we park it parts a token stream that should be all of these basically all the stuff that's in content So I think token stream is going to give us the right thing here Great, right. So this parses out, uh, this gives us back a token stream that is the entire body of the thing Um, oh, yeah, see the same thing nice Right, so we don't really need content because it's going to be the same as this token stream um Now the question is what we do so One way to do this is now just to sort of walk down into this tree until we find Um a relevant thing, right? So this now already parses everything. So we don't really need Uh, we now already consume the content. So that's all fine Um Consider right, we're going to need the two that's fine Great. So now we should pass Uh, zero and two Is our one and two So the next question is going to be for a progress Three So the next one now construct the generated code produce the output token stream by repeating the loop body The correct number of times is specified Okay, so this is going to be Basically now we're going to have to start to actually do the Do the real code generation Repeating the loop body the correct number of times is specified the loop bounds and replacing the specified identifier within the loop counter The invocation will need to expand to a token stream containing. Yeah, so this four times with n being zero one two three um Great Okay, so what this is going to mean is we're going to have to walk the token tree Uh, so let's do Do the same thing we did before so we're going to stick this into main so that we have a good way to see what it expands to Uh, the cargo expand now is going to expand to nothing right the the input code The that we gave this seek if we look at the expanded output It's not there and that is because the output token stream we produce is empty Uh, and that's of course not what we want it to be So here what we're really going to have to do is um interesting So we're going to have to walk down the token stream until we find Somewhere that is the variable Interesting So we're basically recursing down into this is where it would be really nice to have something like fold um But of course the the problem is that we don't have a top level fold. Let's take one more look at fold It'd be really nice if we could use fold for this um fold Ooh x what does x do? Uh, it's not what we want. Yeah, we just like have no idea I mean there's fold block I wonder whether um I wonder whether we can get away with parsing this as a block And then just walking the block instead But it might not matter Okay, so we're gonna do two to from from to two Um No, it's not a block. Yeah, I worry that well currently It is a block, but it's like a General type of block. It's not an expression block, right? So I think it is a block. It's just not an expression block The the real problem is whether it will allow us to also get at macro syntax deeper in the in the token tree So I think we might just have to do this manually So we're gonna do Expand tt with i We're also going to Sim block is a valid rust block Hmm. Yeah, if a if a sin block needs requires that it's a valid sin block, it wouldn't be Actually, let's do this for now. I'll pre-empt myself um So expand is going to take a proc macro to Token stream And it's going to produce the same thing So it's going to get stream probably clone and it's also going to have to Take an i might not know it does need to return the token stream, but it might not need the question mark Um, and here what we want to do is Oh interesting We need a we need a way to basically construct up the output. We're gonna get so I think what we want here is Uh, let expanded is From to map i to this Uh, and then we are going to Well, so actually really what we should do here is at this point. We've already parsed out all that we need um So really we should sort of stick that into here and then implement Uh, essentially implement print for sec macro input. It's not really print. It's like into token stream Right, so really what we want to hear is input into token stream For sec macro input Right, so this is going to have the fields From which is a u size to actually let's make it i size. We don't know if they're negative And Token tree it's going to be a proc macro to token stream Right, so at this point We can do sec macro input and we can produce this thing at this point. We're sort of we're kind of done parsing Um, it really depends on how we want to define parsing I don't think we want to parse any more than this So from to tt and then it's only really at Here when you want to turn it into a token stream That we want to do the expansion You can pick stuff from sec macro input in Inside the sec function Not sure what you mean by that But regardless This I think now in theory. We should be able to do input dot into Instead of that This is going to expand those Yeah, I mean that was basically my my plan was to now do this right quote of Although I sort of want We also need to use quote quote I basically want this to be So where I can iterate and quote I forget It might be that the way we have to do this is Out it's going to be an empty quote as a I guess a token a proc macro to token stream empty and then This doesn't need to collect we just do for i in this and then we do Out is quote Man, I forget from our other one. That's what it is Expanded is expand tt dot self dot tt dot clone i And then this is going to be Out And expanded And then we produce out simpler parse to quote quote I don't see how that helps um Okay, so this is going to basically Keep appending to the token stream what we get from the expansion and then eventually return when they've all been expanded So now the trick is going to be to implement this right and here um This is where we're going to have to recurse into the the sort of depth of the token stream um And I think we're basically going to end up with sort of an expand inner that's going to be expanding token trees instead because I think what this is ultimately going to be is um It is just going to walk the element of the the elements of the token stream So while Actually, let's take a look back at token stream Token stream so token stream implements into iterator. So I think what we're going to do is just walk the Walk this iterator and for each thing Basically look for the identifier that we know we're looking for. Oh wait, that's also true. I need the identifier in here Uh the ident Which is going to be a sin ident So this is also going to have Because that's what it is So we're going to do stream dot Into iter map This is going to be a for each token tree. We're going to do some expansion And then we're going to collect it into a token stream again Because that implements from iterator token tree Right, so we're basically we're going to walk every Uh every token tree we find and see whether we need to expand it Uh and here Uh I think really what we're going to need here is something like Expand I guess expand two for now It's going to take a token tree and it's going to produce a token tree token tree And it's going to match on the tt And if it gets a If it gets a proc macro two token tree So what are the different candidates here? So the candidates are group ident punt and literal So if it gets a group Then it might have to do something. It has to recurse into the group What is a group here? Right And if it's anything else, then it can just return that token tree If it's a group Internally contains a token stream, which is surrounded by delimiters Right, so here really what's going to happen is We're going to have to do Proc macro to group And I have to do this Right, it's going to produce a new one with the same delimiter as the old one Uh and the adopted token stream is going to be expand um So this is calling the stream expand of The group stream And still the same eye And we probably also want to do Set span it's going to be g span So this is going to be the expanded So this is going to produce a This of expanded Right, so this is going to map a group into us expanding the stuff inside the group Uh an expand is just going to call Expand to I guess actually it might have to do this Expand to of the tt with the given eye Now this isn't quite true, right? We know that if we encounter the identifier that we care about the the variable, right? Then we actually do want to expand that differently. So in particular here if we encounter an ident An eye Where it is item we want right ident Probably ident we want um And if that I If that I is equal to oh, so this is still going to have to take self And this is going to have to take self which means they're going to have to be moved into Imple seek macro input self expand um Right if the eye is equal to the Identifier that we were given by the user then we do something specifically then we expand it to Um then it's going to become a proc macro to Token tree literal instead And specifically it's going to become The a literal that's going to be the current eye I guess eye here is going to be an eye size We decided and that literal I think there's like sin lit from eye Although I think we want to make sure that we keep the right span here. So the Lit is going to be this So this is the the literal we're This is the literal we're inserting In place of the identifier in the in the stream And we I think we want to set span eye dot span To make sure that if there's an error with this literal it is tied back to the identifier in the original input And then we give this is let's see what this gives us This is probably all sorts of broken That was broken That is totally true. We're gonna have to do self dot from self dot to self dot expand Love your vim setup got your config somewhere. Yeah, there's a Video in the if you look at the youtube channel if you look at my youtube channel There's a video on my editor setup and stuff. So if you're interested you can go look there I've made some changes since then so arguably I should do another one But it should be there From Oh, you're right. These are not eye size. These are Sin literal Which is kind of awkward, right? Like we actually want to require that they're integers So I think specifically I want this to be a lit int Is there a does this implementer for lit int? Great So this is going to be a lit int This is going to be parsed as a lit int And this is going to be parsed as a lit int. It's still probably going to complain that they're not numbers Is going to be new um, so 45 Lit int Oh, is there a lit u? Oh interesting That's surprising that this is only a u64. It doesn't take a signed integer Well, then Okay, I guess It's going to be a u-size actually Yeah, no, it's going to be u64 Fine It's surprising. I don't know why it requires a u64 mismatch types I expected token stream Dot into Right, this is needs to become So this is a token stream from proc macro 2 and we need to turn it into the actual type that the compiler expects Which is from just a proc macro crate Here Fromson ident Oh, I'm shadowing the eye aren't I? Bent if ident ident Ident Because minus one two three is two tokens. Oh Oh That's awkward. Yeah, you're right The sign integers are modeled as a separate ast involving unary minus Yeah, this might be the a good reason for why to parse it as a range We talked about this a little bit earlier, but it could totally be that we want to use Uh expert range here because Limits That's this this stuff in between From and to are both Why are they box expressions? Right, this is the reason why expert range would be annoying because I guess we would have to check that the experts are the right type So what is if uh an expert lit is a lit is a hmm Wait, that doesn't seem right either Oh, it would be an unary expression. I see Yes, we would actually have to parse it. We would not walk it all the way down to figure out what it is So even expert range wouldn't really help us here Okay That's fine How about now 63 Sorry, you're right. This should say I that was the whole point in the first place From u64 is not implemented for sin lit Uh, really? Maybe like that. Do I really have to expect it start What does am I missing something here? Um Sin lit int Oh, I actually need to do new fine fine fine fine fine and that gives me a lit int Okay, so this is going to be New It's going to take the i it's going to take a suffix Which is going to be None, I guess And it's going to take a span which is going to be the span from the ident Wow, I can't type today either Which now means we don't need this or that And also means we can do this See what that becomes Mays match types expected prokmatro to literal Oh So this presumably there's a Yeah, so lit implements I assume this implements from lit int Uh, and I assume it implements two tokens. It does So Here I could use quote, um But I probably also need Maybe I can just do into here actually come to think of it. I can't turn this into a um Token tree literal So there's a token tree literal That's surprising I mean, I guess I could just construct this instead. I don't need to go through sin for this Um, so I could here instead just do Basically go back to This and say Um The lit is going to be not this but prok Macro to literal u64 unsuffixed Or sin to parse quote Yeah, I mean Not clear. That's even the right thing. It wouldn't give me the span. I would still have to set the span manually Ah, man Why is this? Like really what I want is just this um, but this won't work because Uh, literal needs to contain a prok macro literal not a sin literal There's a macro for that coming Oh for setting the span. Oh quote spend Yeah I see so your proposal here is really that this just becomes Uh sin parse two Um Can I make it just Okay, parse two of quote spanned Uh of span Ident And then oh isn't this quote spanned as some like funky syntax so that so that it's clear where the span is I can never remember what it is though Uh No, sorry quote quote spanned Right. It specifically requires the white space to be like this so this Spanned two Oh, this is the name of it. So this would be like ident And then here I guess it would just be Uh pound i Or on one line with Oh, I see or I can just do this. That's fine Still not going to be right though Uh, I want quote and I want quote spend Actually, is this quote going to just erase all of my It is isn't it? I think we're going to have to be a little bit clever here, which is Um This is going to be This dot map This dot collect Where this is going to be a proc macro two token stream Oh, I see and this parse two doesn't know that this is basically required to be valid syntax So here we can unwrap Quote doesn't require it, but it prefers Oh one line. Yeah. Yeah. Yeah, you can make a veck of syntax and work with that I don't actually need the veck of syntax though. I think this this collect is fine The real question though is Pund ops range Well, that's not right This needs to be this first of all This is now a Can't compare sin identity with this so this has to be that Oh Wait, that's pretty nice That looks awfully correct Right error number zero error number one error number two error number three So I think it generated all of these correctly Oh You don't think we need the into I don't think this would work because expand returns We would have to make expand not return a proc macro two token stream. Don't think that will work Yeah, because expand Um Yeah, we want that to return two so that we can use it in here Which means that up here. Well, I guess we could up here do into Oops here dot into Unclear that the two matter, but then this could be proc macro token tree No token Stream and then this can collect into that Am I confused? This should return This And now into So like it's not clear to me. That's all that much nicer Oh, I suppose I could just make all of you're right. This could just be Uh proc macro two token stream proc macro two token stream Uh Get rid of this Get rid of this get rid of the into Just collect into a proc macro two and then only at the end here The problem is this now is going to be Dot into dot into right it's going to be proc macro two stream No token stream from This dot into Just a little awkward is all All right. Oh, that's annoying to token stream from input into The problem is the compiler is not going to be able to resolve these types for us So I think it's going to mean that I have to do something like Proc macro two token stream is input dot into And then like Output dot into here to turn the proc macro two token stream into a proc macro one token stream But it'll still work All right, so let's see if we pass the test now No, we did not pass the test. We also now have an unused import here Um What is it complaining about one of three tests failed So it expected the output. Oh, it's because I also print the stuff at the top Let's get rid of these don't actually need these anymore Um May end up needing this later. That's not what I'm going to do Let's see So what is it complaining about now It's complaining about the suffix. Yeah, see this is the reason why I kind of wanted to construct this manually because this I think quote always puts the suffix for integers because it knows the type of the input I don't think that's going to work. I think we're going to have to construct it manually Specifically if it wants it to be unsuffixed Right here. It says now error number to u64, right, which is not what we want Uh, so we're going to have to go back to what we had which is lit is uh Soon Actually, can I I can't really construct that with an unknown. Can I I think what I need to do here is just construct this macro to literal u64 unsuffixed I let set span Identity span And then do this Yep And now I don't need to expand either. Oops Great Okay, so that gives us well, we wanted why is it still printing stuff out? Hmm Great, so now we pass the first three Uh, so let's look at progress number four Oh four paste ident One of the big things calls we want to do with the sequential analysis is use them as part of an identifier like f0 f1 f2 Implement some logic to paste together any ident followed by pound followed by our loop variable into a single concatenated identifier Optionally also support more flexible arrangements like f pound n pound underscore suffix I see I see so the idea here is that you might want to use pound n as a thing that appears in an identifier and then we want to Put together those identifiers Okay, so this one shouldn't be too hard for us Right because we already have this expansion business and so here what we're saying is Um If the identifier is just straight up the same as the identifier, we're going to make it n If you find another identifier Uh Where I guess we're going to have to manage identifiers especially here So here we're going to search for pound Ident Followed by self-ident Or pound self ident pound Uh Followed by and uh at the end of an identifier Uh or pound self-ident pound Okay, so if um I guess what we'll really do here is Look at the ident name and then look for any occurrence of pound and see what follows it Uh, so here if we have an ident what do we get from that? Um ident There's a thing here to get Imple Display really is there no way to get the identifier itself That's unhelpful I really can't get the string of an identifier Can I get one with like a sin ident maybe? Ident X it's not very helpful Hmm. Is it really not Is there really not a Ah through the two string method Why is that not listed here? Am I just like blind? This doesn't say that it implements two string, but okay great. So two string I mean you can always format it with display, but you know All right, so let Yeah, but does implement display doesn't give you two string does it or am I misremembering? Yeah Or maybe there's a blanket implementation of two string for anything that implements display Um a two string Uh Name All right, so uh, we're gonna look for Specifically what I want to do is Uh Parts is name dot split on pound Uh for part in parts Actually if parts any actually Is parts dot no string Where the string is equal to self dot ident Unclear that will work I'd guess that depends whether there's a partial to eek for string because there might be Um, so if let sum I Let's do a collect So in that case we're done If it's some we need to implement you need to peek ahead Ident pound Ident Oh because pound wouldn't be a part of the ident Oh, you're right That's kind of awkward I hadn't thought of that So specifically the observation is that if you have something like foo pound n pound bar Right, this won't show up as an ident what I was thinking here was I thought this would show up as an ident That's this right that where that is the ident but that's not true It's going to show up as foo followed by pound followed by n followed by pound followed by bar As we actually need to peek ahead in the stream when we encounter A literal pound Which is also going to be a little annoying because I think It's basically going to mean that we have to do We have to also take the token stream here Which is pretty awkward. Do you need to find a way to stay using par stream? Yeah, you're right. You might be right so in sin Uh, where's our parse the recommendation here was is is there a way that we can keep using Uh parse buffer instead Because parse buffer already has like look ahead methods and such um because if so What we could do here Yeah, so instead of parsing this is a token stream Just have this can be a sin um parse Parse buffer Although I don't think that will work either because this has to be tied to a lifetime Although maybe it's a My guess is this parse buffer is tied to the lifetime Of the input which is kind of awkward Uh Maybe the way to do this is really Not parse the braces here Well, or rather do this Right, but then still keep this as TTS input won't work either Hmm Actually, well, here's what we might be able to do We might be able to turn the token stream back into a parse buffer for the purposes of expansion That might really be what we want to do So down here when we're given the stream, we're going to do this Right we make a new one of these Oh, I can't make a new one of these can I? So this is going to be a sin parse parse buffer And it's going to be stream parse and then Um Actually, why do I I see? Hmm It's a little bit awkward how we're going to do this So for parse buffer Um Into is something parse actually Oh Does it implement just from No I don't think there's an input from here I mean unless I'm missing something here, but I don't think there's a from sin parse to Well, okay, what does? All right, let's take a step back here. What are the functions that I get here? parse tokens of a source code Uh, no parse a proc mark or token stream into something parse is a token stream When I put you a marking I see that gives me a t but what is a parse? Is this implemented for like parse buffer? No I would have to know what type to parse it into Which I don't Yeah, I don't I don't think I can use this because I don't have Um It's not clear to me that I want to parse it as anything in particular Is the awkward part Like I basically don't want to write implements of parse because I would have to parse Every possible item because you never know where the end is going to appear So I would actually rather walk it as a token stream Uh, and I think from memory doesn't token stream also has a have a peak Uh, where's token stream here? I see. I mean it has a clone it has clone an interatorator Which is still kind of awkward Yeah, no, I think I want to keep it um Think I want to keep a token stream And then really just do Yeah, here's what I want to do Hmm It's a good question What is being coded? Uh, we're doing the second part of the procedural macro workshop from rust ladam I recommend you look at the first procedural macro stream that we did a few weeks back if you are just jumping into this now um Chat needs a way to pin something What were you thinking of pinning? Um, that's kind of awkward because we don't have a good way I think what we have to do is give We have to give expand to access to the following parts of the stream So what does the into iterator give me here? It gives me one of these Oh the answer to what this is about. Yeah, we should pin that you're right um Yeah, I might have to clone the stream Which I think is fine. It's just I have to clone it At the current point is where it's awkward I guess I could clone it just every time I call expand The recursion is going to be annoying too though Because really this is going to be like rest, right? So if this is rest Then now This is fine because this this would have to also give rest and expand What am I talking about and yeah, because No, that's right G stream Yeah, no, that's fine. This this only needs rest because it's going to have to walk it there So all we need to do is find a way to give The rest of the token stream to expand To expand to I mean and I think the way we're going to have to do that is um Is definitely pretty awkward I basically want a way to walk the stream without Wait, is in is this clone if that's clone that is clone. Okay, so this is far easier actually. I'm just being silly um So the token stream is going to be stream Into iter Here this is going to be far easier Where's my this is Token stream So here what we're going to do is while some tt Is equal to I guess lead newt prog macro 2 token stream new Uh while tts dot next Then we're going to do out dot uh extend self expand to Uh tt and tts dot clone Ha even that's not going to work because um We needed to return how far it walked I think Oh, so maybe we Maybe we just do this Yeah Yeah, yeah, yeah. Okay. This will this will be good. This will be good. I have faith Uh, this is going to do this and the I and then this is going to return out And now this has access to the rest mutably so it can choose to consume more tokens And so here this is going to be all sorts of funky So if it's that that's fine. It doesn't have to do anything special Here if it gets an identifier It's going to have to peak and the way it's going to peak is by cloning the iterator Um And then Actually, it's not even if it sees an identifier It's if it sees a Oh, no, it is if it sees the identifier because the where we where we get into trouble is there are a couple of things we could see, right? We could see foo n bar We could see uh foo n. We could also see n bar No, that can't be the case because the identifier can't start with numbers. Okay, great so you will always see an ident first and then uh if And then I guess what we're going to do is Match peak dot next Right if the next thing is a proc macro to token tree literal Uh lit If the literal is equal to this Then something uh Otherwise If it's not then we know that what we just got was an identifier that was not followed by a pound It was followed by something else and in that case we don't need to do anything about the identifier We can just leave it the way it was So in that case we can just do this Which means that we can just do Yeah Okay, this is good. This is good. This is good. So now Um, if we go in here now, we need to sort of search further. So the stuff down here is probably just wrong So I don't know if I can do this comparison, but it'll be fine regardless. Okay. So at this point what we've seen Have seen ident Followed by pound So the question is what follows this, uh, is the next thing the chosen identifier Or the repeat identifier Um, and so here we're going to do Uh match peak dot next Uh, and if that is a proc macro to token tree Uh identifier Uh Ident two If ident two is equal to self-ident Right, if that's the case then we have seen ident pound n We might want to clean up this a little but for now i'm just going to leave it this way Okay, so now the question is what comes next so match peak dot next If it's none, then we know that we should expand Because that means that the pound n was the last bit in this block um If it's some Actually, should we just always do it in this case? It might immediately be followed by some other identifier Right, so imagine that someone writes like Fn foo pound n I guess it would be this so it would be a literal open parentheses or it would be a literal group actually no it would just be a group But you could also imagine something like const No Can't actually think of a thing that it would be Uh, yeah, the question is in this case In what case would we not expand the n? I think in this case we always expand the n the question is why does the exercise talk about uh If you remember back to the exercise The exercise also talks about pound n pound oh because It needs to be because otherwise the n would be n underscore suffix, which we would not recognize as n I see so it's specifically if um, if the ident is this Then uh need to also consume Another pound You don't have to worry about that because at the end rust will still validate your output You can just transform into a new identity move on Yeah, but we do need to consume if the if the user wrote another pound after the identifier We do need to get rid of it because otherwise it would be invalid syntax So we need to consume it from our input so that it's marked as consumed because otherwise rust would say unexpected token pound I think Specifically because we replicate all the input we get that we don't specifically get rid of Um, we would just repeat out the pound sign as well, which is not what we want to do And so I think here we want to do Um, ah, how do we even do this? um Okay, let's leave that for a second in this case What we do know is that we need to produce Uh, a new identifier Actually a quote spanned, I guess Uh, it's going to be the spanned uh the span of ident And it's going to produce Um, it's going to produce ident I Oh, I guess this has to be sin powers two. Let's just see what it gives us for this Well, first of all, I did something silly somewhere Here and I also did something silly down here. I probably also did something silly somewhere else Like on 67. What did I do here? Found colon oh line 71. Yes that also means one of those six. Yes indeed 72 can't compare Literals Of course it can't Of course it can't But this we know would be a pac mark or two token tree Lit otherwise it couldn't be a pound That's not true. I'm totally lying um Actually it would be punctuation. It wouldn't be literal if the next thing is a punked And it is a what is a punked if punked Is one of these guys pound is not literal string. Yeah, that's okay. Yeah Um, so let's see Oh, am I really not allowed to compare that? Fine if punked as car 76 should say oops Should say ident And what else do we have 79 should say? Uh right this we can unwrap I think Token tree is not an iterator eight a second What else does Where is my token stream token stream to implement Yeah, extend token tree Ah fine iter once Needs to be an iterator, huh? Uh 76 it's complaining that I moved something That's fine. I don't particularly care if it doesn't move that Two that's fine. This can also be by reference Uh 81 That's fine. This can also be a reference Don't actually need to own any of these 83 That's why I wanted to own that Fine not important probably Oh, I guess our main needs to look like our four For this to work out Let's expand to proc macro panicked Ooh unexpected token Where did it panic? Though also why is it? Oh, it's printing that just to be helpful Okay, let's have it not print that But where did it panic is what I really want to know Unwrap here, maybe we made invalid syntax Yep Pound is not Litster Don't think I'm assuming pound is litster anywhere Line 80 or something Yeah, it's here It doesn't need to be litster Um, it's saying unexpected token, huh? Um Macro parse to parse to Yeah macro Not sure what you mean Felix Um Great, um Yeah, so apparently it doesn't like this it doesn't like trying to parse This ident that we create here. We might just have to create the ident ourselves Um Which is luckily pretty easy, right? So um going back to here Where's our ident? So we can just create this with new and all we really need to do is give the string in the span Um, so here. We're really just going to do proc macro to ident new And then here we're going to have to construct the string that's going to be the be the ident And that of course is just the ident and the i concatenated to one another and the span is going to be the um Original ident probably I think that's the span we want That way we don't need the unwrap either And we no longer need quote span Uh line 81 Token tree found ident Oh, I see I think that means I can do this instead and then I can here do Ident is equal to that Do this I could do this And down here I could just return that How does it like that? Hmm That doesn't seem great Expected one of parentheses or square found pound Oh, right. We're right. So here Uh, what's happening here is basically We are still emitting the pound symbol right because Um Here we given the iterator But currently because it's just peaking ahead it it doesn't Change the iterators or it doesn't skip the pound sign when we find it here And so once we get to this point we need to do Uh, we need to set We need to set rest To be equal to Uh peak How about that f1 one times two f2 two times two f3 three times two Not bad not bad not bad at all We still have this to do right like it might still be that there's yet another Um, but there's yet another pound, right? So remember the case of uh foo Two pound and pound bar currently we wouldn't we would correctly replace this With the integer, but we would not get rid of this pound And so here, uh We may need to also consume another pound Uh, and so if I guess we're going to do peak dot next The problem here is Uh, actually We might need to if we peak ahead So if we peak ahead and it's not a pound sign Then we don't want to consume that next token. And so this is why here We actually need to take the current peak and store it into rest and then continue operating on peak But we don't have a way to put something back into an iterator. So in this case We I think really what we want here is this um Only in this case And in no other Uh Do we want to do this? Right if we found anything else Then we don't want to touch the we don't want to set rest to be equal to peak again because peak has consumed the next identifier All right, so let's see how far we get in tests Is it happy with our test for it is happy with our test for excellent I also really want to get rid of all these match statements Um Sadly, I think it's a little tricky There's a there's work on getting If let to work with uh, and which would make this a lot nicer Uh I think in theory we could at least do this a little better by saying peak dot next peak dot next and do Do this this To this If that and this to that then this And now we can get rid of some of these Am I missing something probably Yep That's a little nicer. Let's see that it still works great All right, so progress number five Repeat section so far our macro has repeated the entire loop body This is not sufficient for some use cases because there are restrictions on the syntactic position The macro invocations can appear in for example, the rust grammar would not allow the caller to write That oh, I see you can't call a seek you can't call a macro here Um, right exactly instead we will implement a way for the caller to designate a specific part of the macro input to be repeated So that anything outside that part does not get repeated the repeated part will be written surrounded by Uh pound open star the invocation below should expand to whatever. Yeah Optionally allow it for there to be multiple such section. Well, the test suite does not exercise this case Each of them will need to be repeated according to the same loop bounds Oh Uh, are you planning on a break? What kind of break a real world break? Uh in what sense? Like during the stream. I wasn't planning on it. I have my water. I'm happy Okay Right, so now we're gonna have to be able to parse something like this um So this okay, so this changes the calculus a little because now Um Now this is also funky because it means that we need to We need to figure out whether there are such repeat sections in the macro body Before we decide whether to repeat the macro body or not Um interesting interesting interesting interesting Okay, so we're gonna have to walk the input and if we find a pound followed by a group Right a punk pound followed by a group followed by a star In that case we want to repeat the body multiple times And in that case we don't want to repeat the overall body of the macro So This is now going to have to shame Because I think the way we want to do this is we probably want the I think we want to try to do the inner replication first And if there is no inner replication then replicate the outer I think it's going to be the plan Okay So in that case So expand is going to Interesting How do we want to do this the the problem is we can't just Run the inner and have it like set a boolean as to whether or not it's expanded something because If it told us it didn't expand anything Then would it leave what would it do if it found the the repeat identifier that has to be replaced with a number I guess one way would be to run The expansion in like a mode where it knows not to replace the identifier Oh, that's kind of cool. So it will basically be like a two We can do this in two passes actually the first pass Uh Does the inner replacement and the second would do the outer replacement So basically what i'm thinking is first you run a pass that uh expands any inner iterations Like any inner like pound bracket star And if they did something Then Then you just return the resulting stream If they never made any changes Sorry, and they're they have to not be allowed to expand a pound n They have to not be allowed to expand the identifier to the number unless it's inside of an internal repetition Yeah. Yeah. Okay. Uh, this will be easier to explain with code. Um Okay, how do we want to do this? I think we want to do this by saying Uh expand Ident This also means that there's going to have to be this is going to be expand Pass This is going to be We need to give these better better names, but let's write the outer one first so Imagine that we have this expand that is sort of the the overall procedure We're going to follow and that is going to here do Um out is self dot expand pass stream dot clone and false And then It's mutated Uh, if mutated Then return out Otherwise self expand pass Not clone true So when this variable is and then let me fill this out. So, um Let me mutate it is false Um, this does not have the eye. Oh, this is weird. Um, okay, so Hmm, I don't know if this is the best way to do this, but but the idea here is going to be that expand to When expand ident is false Is only going to only going to expand repetitions Nothing else Um, let me see if I can make this a little bit more explicit. So we're going to have an enum of like, uh, let's see mode Which is either going to be replace ident of Which is going to be the ident Uh, I you're going to replace it with or replace sequence And so it's going to be in one of those modes So this is going to start with uh, mode Expand What did I say what did I call it? Place sequence Otherwise it's going to do Uh self dot zero uh self dot from self dot two span pass mode replace ident eye This makes sense. So it it's first going to do a pass where it's not going to replace any identifiers It's going to write out the tokens and then replace only sequences If that indeed changed something Then uh, then we're just going to assume that we're done Otherwise we're going to walk the range and then replace the whole thing with the identifier replaced This means that think of this as something like what if a user writes In zero to ten Like if they write something like f n x n something Uh, and then they also write f n y n Don't know why they would write this but imagine that they wrote something like this Um, what should this expand to right because this outer one sort of implies repeat the whole block n times But this part of the block is saying repeat n times And so if if we did in fact repeat the whole block n times then what this would end up being is This line n squared times Right, which is just not what we want it to be. And so I think we're basically going to require that the user gives us one or the other specifically If we encounter code like this the first pass in which it's only allowed to replace sequences should emit Uh an error when it hits this Right, that would probably be the best way for us to do this to say if you encounter this, uh in Oh, no, we can't even do that. It can't be an error because then this would be an error because the first pass is always going to be the same So So if you see code like this, what should we do? um The first pass is going to replace all of these I think the user is just going to see this as a compile error because this is just going to remain pound n Ideally we would give a slightly better error message, but I think that's okay So this would actually expand only the innermost one and not the outermost one And that's basically what this expand pass is doing, right? It's first running a pass with only replacing sequences And if they did something then we consider ourselves done. Otherwise, we're going to replace the whole block with every i Okay, so let's try that How well maintainable are proc macros? Well, what do you mean well maintainable? Uh, this is not python. This is rust the rust programming language It's funny that rust gets mistaken for python though So arguably a good thing for rust Okay, uh Let's see So This first pass It's going to do a normal pass the way we did So I think all of this actually Stays the same We just pass in the mode up here We also need some way to express mutated Um Mute so this is going to take rest mutated bool And a mode If we get a group We're going to this is going to use expand Uh balls, what's that even going to do if it gets an ident? So in this case We replace something so we're just going to set Mutated as true Also, we only want to do this if the mode Let's us replace the ident So if Let mode Place ident I Is mode Then we do that otherwise We're not allowed to replace this ident And so it's going to stay what it was I guess ident dot clone allowed to replace items in top level pass In First pass This is not for a game. No, this is not related to the rust game Okay, so That's fine This is going to have the same restriction where Um, let i is equal to that then i otherwise Otherwise return that ident Also, this doesn't why did why did I even do that? Shouldn't be necessary Um, then we're just going to return the ident here Because technically we can do this. It's a little less nice but fine See we could do even better. We could say Mode here and then match on it down there that way. We don't need the a flat Right. So if we're in the replace ident and we find those things That's all great So now there's one more bit we need here Which is what happens if we find a um If we find a proc macro to token tree punked if p dot ask car Is equal to pound Is then we need to figure out whether this is a repetition sequence Right and now we're going to play the same trick like we did previously Which is basically this if And then we're going to match on peak dot next twice Because we need to look at the next two items, right? And if those are If those are a group So I guess the group would be their petition and this would be the star um So if what we find is this if star dot as car Is a star Then yes indeed actually and Not just that but also the repetition delimiter the group delimiter Is this Right so so this case is we got a pound followed by a group followed by A punctuation where the group delimiter is parentheses So I guess this should be And the punctuation that follows is a star The other thing we might want here is we want to make sure we're in the right mode So here I guess If we only want to do this expansion If we are in Replace sequence mode. Oh, you're right. I should say star Good catch Although this part is going to be funky because here Uh, we're going to have to return a token stream not a token tree I think Bomatic I'm going to have to figure that out in a second um Actually that might be an easy change Otherwise, we're just going to keep it the uh same thing it was So if it's this Then now Funky things to do Here we're going to have to repeat the stuff that's inside. Luckily, we already know how to repeat groups, right? It's basically what we did up here when we encounter a group earlier Right down so if we go here Down here Right, uh, we're going to expand it into Uh, we're going to run. Where is it here? Expanded we're going to call expand pass. Let's ignore those bits for a second Yes, uh expand Uh as many times as uh for each Ident for each L for each Sequence in the range The problem here is this has to return a stream, right? This is not we're going to replace the contents of that with a sequence of things basically a token stream Whereas what this function currently returns is a token tree So we're going to have to make this a token stream Luckily, that's not too hard to do I should change the stream title to what's going on Yeah, this is one of the things I think is the biggest pain with streaming is like I don't want to have to go through all of the different channels that I multi stream to and give a title each time But arguably I should Um Usually like my announcements of the events happen on twitter anyway I sort of want all of these to just be synchronized But yeah, arguably I should So I uh, the real question here is whether I can easily get a token stream from Um Basically Ah, here's what we'll do We're just going to assume that it's a token tree And then we're going to do Iter once TT collect Because now we can early return in here with a token stream that way the other code doesn't have to change Uh, right. So this expand pass is actually not going to quite going to be this What it's really going to be is Oh, I forgot we also need to do this like mutated business This also mutates This is also going to mutate So here Here we're gonna we've encountered a sequence So we need to expand it as many times as are necessary Which means we basically need to run the expand pass on this sub stream with the mode that is replacing the identifier So how's that going to work? Well, it's going to be pretty straightforward actually so from uh, value self to value map I expand mode replace Ident this collect Which is collect here is going to give us the token stream the concatenated token stream I think in theory that should do the trick um Oh, this should be wrapped on stream. You're right. Thanks. Good catch um Where this is going to be a little bit tricky potentially though is uh Can I set the span of a token stream? I don't know probably not Might not matter. Uh, what else do I have to do here? I also have to set rest is peak Because we've now once we've done this then we have parsed up to and including the star All right, this is probably going to be all sorts of broken 65 star ident four delimiter This should be a proc macro to the limiter what proc macro to delimiter Uh parenthesis We might not even care that it's a parenthesis, but you know might as well do it properly Uh This should return out and mutated So this and a pool This is going to then map out The token stream because we don't need the bool here expand pass This also ignores the bool And 162 Uh, this should also ignore the bool 112 Really? Sad, but you're right. Oh, that's not what I meant I'm really gonna have to make all of these be that Pattern underscore not covered sure Uh an underscore Is it gonna mean do nothing? So we're gonna go back to making this a mute p Um, actually it doesn't need to be a mute p even it's just 65 This really have to be that So frustrating mind by move p needs to be a ref because Ascar takes self She does ask her even take self. I'm not convinced Yeah That doesn't consume self at all All right use of mood value mode. Oh, uh, that's fine mode is all of the things Uh, it is Copy it's clone It's debug Probably other things too, but you know Oh Okay, so let's let's first see that we still pass the first ones Great. Okay, so we still pass those now. Let's take a look at five So five, uh, if we go back to our main See what that expands to Oh, that got some errors Uh, expected square bracket found parentheses Huh Oh, that's the compilation. Okay, so what did we generate for this? We generated nothing I see So that's probably us generating this message We expected square bracket found parentheses Did we now? huh All right, so something is wrong So let's see how far we get here first pass second pass See whether we even get to the right pass So it Oh, that's interesting. So it runs the first pass and the second pass and that is certainly wrong So the real question is why does the first pass Not catch this found square I'm assuming it's going to hit that a number Wow It hit that more times than I thought it would Sound things weird should not have found it that many times Okay, well I guess before And then here we have the luxury of being able to use rest Before oh token re-iterator. That's annoying Founded before derive Founded before derive Founded before derive Why does it call this so many times? first pass So confused Oh, because it runs the second pass. That's why I see so really Let's just to make our lives easier oops macro two Let's make sure it doesn't actually run the second pass So that we can write So it found it on derive and it didn't find it anywhere else So the first pass did in fact correctly Replace the thing but mutated was false Oh, here's an observation. This should probably not do that anymore So somehow it All right, what does it look like after the first pass? Expand here after first pass it looks like this After the first pass is just a pound That seems wrong Let's extend. Oh, I wonder Which pound did it find? So it found the derive pound And then after the pass Is that the same Spen The bytes are a little off 37 30 no 37 38 Okay, so After the first pass the only thing that's left is the first pound So the the one that's before the derive So this makes me think that that That this code for some reason ends up replacing the entire stream with just the first pound symbol it finds Which seems not quite right. So do we even get in here? time to replace I'd seriously doubt it, but you know We did not So it gets to here So that's a punk expand. I guess we're gonna have to Expand tt now have out in fact, let's do Expanded This extend expanded expanded to that Let's see what this actually expands to when we run it So it expands The pound and it expanded to pound So that seems correct. But why does it stop iterating? Why on earth does it stop iterating at that point? Because the stream we get in should be the the full program stream like if I do this Presumably that's the full program It is not it is just the very first token Wait, where does stream come from?  What is this token tree? That just seems wrong The token so the token stream that we get when we parse the stuff in the braces What is content and what is tt?  Yeah, this just seems wrong the content It's saying that when it parses The rest of the input after the range With braces the content is just This actually just this symbol Which is just not right Because like if we go up here What is input at this point? Oh wait, and then maybe I'm just reading this wrong I'm totally just reading this wrong all all the characters are there.   Okay, so we do actually I'm being very stupid so found pound before the derive Right, so the rest of the stream still has things like the star that we want to expend Okay, so it found pound before It's not in that It is in that mode So in theory it should go in here peaking it does go into peaking But it doesn't go into here because it doesn't find the appropriate pattern So it returns the punctuation which is correct and then it returns The correct token tree. That's what it expanded to then it extends out with that So the real question is why does this not do another iteration? After that expanded to Yay, I know about debug. It's just in my fingers at this point. The debug macro is really nice Uh, also gotta use an And else you move the thing inside the deep. Oh, yeah, sure. It's fine um So the real question is why does it stop? Consuming things from the token stream. Why does it think the token stream is empty? Does the peaking not actually work? Okay, let's first of all make this clone rather than consume the rest of the input Great much better Okay, now what did it generate? Okay, so It's walking the stream Founds a bunch of derives That's fine. It walks some more. It found found enum found interrupt Then it finds Brace So it finds this square bracket Which is this brace Um An inside of there is the pound the parentheses token stream that contains the stuff where we need to replace things um Cointuration Found pound before I see what's wrong Hmm, I see what's wrong. No never mind. I don't see what's wrong um, okay So It found a pound before the Correct thing it decides it's time to replace. Okay, so it correctly identifies that it's a pound followed by a group Followed by a star So now it doesn't expand on the inner bit. So it expands irq to irq zero Followed by a comma It replaces irq with irq one. So all of those are expanded correctly So the full expansion of that pound is irq Zero followed by a comma rq one followed by a comma. Okay, so all of that is correct. So it correctly expands that expanded to so Overall that group expanded to a brace so this is the group that starts at here so That expanded to a group with delimiter brace. So that's the variant list Followed with the contains a stream that has irq zero irq one irq two etc So that seems that all seems right It just seems like it doesn't It doesn't realize that it's done It's because the mutate happens inside of a group and this doesn't propagate that change to mutated. Okay, so This is going to have to be this Immutated Is this And then mutated Equals g mutate. Yeah Now we're talking. So now it expanded to all of them beautiful Right, so the problem was this wasn't propagating out the mutated to get back from expand pass and therefore This call had mutated to be false. So it thought the first pass didn't change anything. So it ran the second pass Okay Nice, okay progress So number six Make work in function as the first 134 function like procedural macro calls are not supported inside of a function body by the stable compiler What do you mean inside of a function body? When you enable this test case, you should see an error like this Cannot be expanded to statements optionally Okay Great. Yes. I did see that error. Now what? um Optionally, if you have a nightly tool trying to stay entire temporarily adding the following features to this test Okay, to this test case But before you move on let's fix this in a stable way Check out the proc macro hack crate for a way to make this work with a relatively little effort Shouldn't need to be any change to the macro implementation for this test case beyond picking up proc macro hack before the completeness the expanded code here will look like That interesting Oh, what a surprise the Workaround is also written by david Great ones contain nothing but procedural macros Private helpful for some private ones are fine, but nothing can be public. Oh, that's so funky Shouldn't need that Hydrogen instructor says the expanded code can't refer to local variables other than those past Micro must refer to local variable fake call, but why? I mean It's not very hard to fix I suppose, but why is this fix even necessary? Okay, so We don't need these anymore You uh So what do I need to do I need to do this and then make This no, where is it down here? It needs to be a proc macro hack instead of a proc macro Does that just make things pass? Oh, right. I still need extra and create proc macro because it's special Proc macro is the crate that the compiler exposes which apparently still requires unresolved imports Well that did not work interesting so now Seek is not being exposed at all so this Really makes it seem like proc macro hack does not work correctly Don't you need at least two crates? Oh two crates are required. I see The implementation and the declaration crate That's awkward Oh, I see That really is a hack Okay, so That means we're going to Uh cargo new lib sec proc move source lib to sec proc source lib remove copy cargo tumble to sec procs there and my This And here we need a source lib bar. What? Yeah, that's fine. Uh, and that's going to have this So this is going to have a sec proc which is going to be equal to path sec proc and this should not have that right And this Should be like sec proc And this needs no other dependencies or something Actually, it needs proc macro hack, right and also macro expansion ignores token Curly brackets that seems even more broken. It should not be necessary So I am running on beta currently, so I don't know if there's something on beta that would make this not work Macro expansion ignores token curly bracket and any following in the Wait in our main Am I missing something from this? like Macro hack Yeah, and this does the right thing there This externs this uses They both depend implementation crate Using why? How is this different? Let's see. Is this a known issue? One closed Rust macros in expression position must expand to an expression I see so this is really saying So this is the claim here is that this is a bug in our implementation That we have to produce something that's valid in expression context What does this test case say? Shouldn't need to be any change to the macro implementation for this test case beyond piggy on a proc macro hack huh I mean macro input I'm not sure I understand what it's complaining about because in this case um Engine ignores I mean this seems like exactly the same Error expression position must expand to an expression So this suggests that it it thinks that what we're generating Here Isn't valid wait. Let's see exactly what it is we generate so here Uh, this gives me a Z equals that fine no, actually I specifically want this Which I think I can this is legal How does it crash before I even produce any output though? Right like this seems to complain about the input not the output because Uh, I don't even get to the point where my output happens here Ignore some and any following I think this is a test case bug fixed in master Okay, uh, let's do a Get Get fetch upstream Okay Doesn't seem like it Does it need to be fixed in earlier test cases too? Yeah, because I think I think the problem is the earlier ones Let's see New text keep the original seek macro for use outside of function bodies introduce Oh Oh, that's funky. Okay. Um So slip so this You're telling me that what I should do here is have This be a proc macro And this be a proc macro hack this and then Source lib this guy is going to use I guess this And this Cargore might not need to attach that one Okay. Yeah, great. So this is basically keeping the old one the way it was But introducing a wrapper for The test case Okay, no, no, that's fine I mean, it's nice that it's nice to so we we have david in chat and It's very handy that he could just fix the test case while we were running into it. Um easy So should it be easy graph seek? I guess easy Did I just like read it wrong? Did it say? Yeah, it says f seek I see so this text should say f should say easy Nice. All right. So cargo test All right, we have six make work in function Progress All right, seven. What do we got in seven?  Okay Fine david. I will implement inclusive ranges If you insist Okay, so this should hopefully not be too bad for us. Um I think actually we're gonna change the orientation here a little bit and Rather than store to and from here store, uh range Which is gonna be a rust ring. Isn't there a Really? Is there a separate type for inclusive range Didn't think there was Yeah, there's not to from an inclusive Inclusive Is the range inclusive? So the real question here is how do I optionally parse a token? I don't actually know what does parse say? Aha, I'm guessing it's this one I'm just gonna let inclusive is Token dot Actually, it's gonna be option token dot parse input Is some yeah, that's that's sort of what I guessed Uh, let's fill up our main again And expand inclusive. Oh, so many errors Expected literal Did you expect the literal? I don't think that's true. Oh, sorry, this should go before two, of course Now written as dot dot equals and you should swap the order Inclusive ranges are now written as dot dot equals. Yeah That's what I'm parsing Um But why did it expect expect a little oh, it's a specific I see so it's it's really that I have to do this It's it's considered one token Dots one Exclusive Inclusive It's a it's not Three tokens. It's uh, and then I guess I want to assert If not I think really here's what I want. I can't option that Why can't I option that? Oh, I guess I could just parse it as Fine, I'll I'll I can pass it and parse it as a range. Fine. Fine. I give up I give up I give in Uh expert range fine fine fine fine fine I will parse it as a range Okay, fine fine fine. Why not arrange? Wait, what do you want me to do here? You want me to use peak? so Okay, fine the I see so I can't What does uh parse give me actually parse buffer peak I don't have a parse buffer. Oh, I see from parse stream I see so what you really want me to do here Is with the parse buffer Do a I see so here. This is going to be input dot peak um token Dot dot equals And if not inclusive Then Just to give a better error message We should be able to do token parse input I don't think this will work because I think this is going to generate a type expected literal Hmm. Oh, I see what you mean. Yeah. So we need to I see You see this is what you want me to do Except this would be that and this would be this because we need to consume the input regardless right Hey And I still want this because I want the variable And I want the This and this Oh, right. We're not actually using the fact that it's supposed to be inclusive. Great There are two places I iterate so let's make an actual iterator um It's going to give me You know what? So because we're operating on numbers Um We can do this Normally, you can't do this normally. These are not the same But in our case, we can uh, except this has to be value value Value And and then this is going to be self-range And this is going to be self-range Some big pyramids Yeah, these are kind of unfortunate Um It would be a lot nicer if you could use and and if let's I mean arguably we could like Split this for example into a function But there's a lot of like local annoying stuff for we're gonna have to pass all the arguments along All right progress Nine what do you got for us now? Procedural macro API uses a type called span to attach source location and hygiene information to every token In order for compilers to appear underlining the right places Procedural macros are responsible for propaganda manipulating these pants correctly The implications below expand to code that mentions of value missing zero, which does not exist When the compiler reports that cannot find value missing zero We'd like the error to point directly to where the user wrote missing pound n in their macro aim But for this test to pass ensure that the pasted together identifiers is created using the span of the identifier written by the caller If you're using the nightly toolchain, there's a nightly only method called span join Yep, so that we would get a disband including the pound n Yep, I think this is just gonna pass I think we've been good enough for that to happen Yeah Why don't you use the inclusive range in the range method So the reason I don't do that if you take a look at it fn range So impulse iterator requires there to be a single concrete type that implements It's not a din, right? It's an impulse And dot dot equals returns a different type. This is why earlier I looked up standard ops range and see how range And range inclusive are different types. And so therefore I can't return Both like I couldn't do dot dot equals here because these two match These two if arms would have different types. I could return a box Or I could I could do a box din iterator, but I can't do it with impulse iterator all right 10 interaction with macro rules Suppose we wanted to seek invocation Which the upper bound is given by the value of a const Both macros and cons are compile time things. So this seems like it should be easy I suspect it's not going to be easy In fact, it isn't Because macro expansion and rust happens entirely before name resolution That is a macro might know that something is called mproc But it has no way to know which which of many mproc values in the dependency graph does that Refers to Or maybe the mproc constant doesn't exist yet when the macro runs because it is only emitted by a later macro that hasn't run yet And this completion model It isn't possible to support a query api where the macro Can give it the name of a constant and receive back the value of the constant All hope is not lost It just means that our source of truth for the value of n proc must be a macro rather than a constant The code in this test case implements this work around This test case may or may not require code changes in your seek macro depending on Uh, how you have implemented it so far before jumping into any code changes Make sure you understand what the code in this test case is trying to do Okay, let's do it So this uses e seek source of truth Call a given macro passing n proc As an r want this number to appear in only one place so that updating this one number will correctly affect anything that depends on the number of procs past proc procs array So this calls the macro make procs array with the argument 256 So calling make procs array with 256. This is going to be a literal 256. So this will work just fine and this is going to Call the macro literal identity macro with the value 256 returns 256. So this is set. So in theory, this should just work I think nice All right, so This means that we now have all of the things right Pass all the test cases Good job. We did another macro exercise All right, let's do uh get a dot Uh Seek implemented And push that beautiful All right Now What exercise should we do next? Given the time we're at I think we should do sorted because my Theory is that will it will be faster than trying to do custom debug. I don't think we'll have time for bitfield um So I think in fact bitfield might be a good exercise for you as the viewer To do after having gone through this with me. Um, okay, so let's do sorted um So in theory all this does is it's like, um It's almost like a clippy lint, right? It checks that the variance of the enum are in the right order And then emits errors correspondingly Is that a word unclear? All right Sorted, huh? Uh, so let's go to sorted Uh cargo tomo. I'm going to assume that we're going to need sin Uh 15 We might not need quote actually come to think of it. Do we even use quote? I might be able to just remove quote from the other one Uh, all right sorted source lip And test progress First test 01 pars enums See what it says this test checks that an attribute macro sorted exists and is imported correctly into the module system If you make the macro return to empty token stream or exactly the original token stream, this test will already pass Be aware that the meaning of return value of an attribute macro is slightly different from that or derive macro drive macros are only allowed to add code Um, so what they return is compiled in addition to the struct enum that is the macro input On the other hand the macro the attribute macros are allowed to add code to the callers grade But also modify or remove whatever input the attribute is on The token stream returned by the attribute macro completely replaces the input item, right? So this is what we talked about earlier when we talked about, uh, both derive builder and, um When we started on seek how there's a difference between proc macro derive and just proc macro where proc macro derive Adds code that includes like the implementations and the other types, but it doesn't change the originally annotated type Similarly a proc macro if you if you annotate a function with proc macro It replaces that entire token stream with the implementation of the macro instead, right so or so that what that means is the The meaning of the function signature token stream to token stream is different for the two for the Custom derive it's Here's the token stream for the input now produce a token stream to add For proc macro. It's here's the input token stream. Here's return what to replace it with and in the case of Attribute macros. It's the same it also replaces The original input item so it's more like a proc macro than it is like a proc macro derive Uh Also parse the input token stream as a sin item in order for item to be available. You need to enable features full Uh, this is by default. It only builds the minimum set of parses needed for derive macros great So Let's do that. Um here we're going to use Sin Actually, what even are we going to use here My guess is i'm just going to bring in like the the other things that we know we probably need so proc macro two is version zero four um And we're probably going to need Actually, we might not even need anything from sin for now um So recall from the previous stream that for sin there's this like super handy dandy proc macro input macro Parse macro input macro um where we do like this parse macro input args as uh Sin args And item Specifically, we could probably parse it instead of just parsing it as an item We actually know that it's an enum, but that might not matter I don't know if there's a an enum item Probably not actually So an item enum Great, let's just parse it as an item enum instead So this is going to be our Our type And this is going to be any args disordered and we sort of we assume that it's that uh We assume that args is going to be empty All right, so here let's go into sorted cargo t actually is do cargo expand first Huh Did I do something stupid? Yeah And of course, we also need to use sin parse macro input Sin args in meta Builder source lib. What do we parse that as? Oh interesting interesting Uh sorted source lib Into does not exist, huh? two tokens I guess what you're saying is also bring in quotes And use quote two tokens I guess we could do that, but it seems like a waste Expected proc token stream found proc 2 great Uh, I'm on a debug of this And I want a debug of this Unexpected end of input expected ident Oh, it's not a meta is it? Let's see what args actually is But We also what what did we say we needed uh features to be extra traits And full it's an empty tokens stream Maybe it's just adder I forget what is so derive input is a Thick of attribute I think it's a met I feel like this was meta like but okay So we're trying to figure out how to parse the the pound sorted in the first place and I feel like this was called a meta Meta list Something like that right right either Meta Yeah meta So why can't I parse that as a meta? Yeah, I met as a word like test in that great Uh, but it's not letting me Do that. Why doesn't it let me do that?  So for now, we're just going to assert that args is empty and then we can get back to that later But for the time being cargo expand of that just now produces the enum that we originally had great and we parse it So we should pass the first test Beautiful. All right Test two so progress Show me test two Sort of markers only defined to work on enum types So this is a test to ensure that when it's attached to a struct to produce some reasonable error Yeah, so we just parse it as an enum item I see so here we could we sort of cheated actually by parsing as an uh item enum because here the argument is really that We should use compile error like we if you remember back to When we did the derive builder We had to basically emit compiler error statements in the output token stream to make sure that we produce the right errors um, and here Really what the intention was was that if the thing you got was not um Was not an enum you emit a compiler error thing that says this thing is not an enum Yeah But we don't might not need to do that theory It expected that we say expected enum or match expression Oh I see so this is what Sneaky so the exercises haven't told us yet that they want sorted to work on match as well And so therefore we can't get away by parsing it as an item enum because It's supposed to also work on match Which means that it's going to have to be an item and that would have given us the the right thing here Okay, so let's do it the way it's supposed to be parsing as an item Uh, and then we're going to basically I guess match on tie Actually, we don't even need to do that if let, um Sin item met uh enum So I guess where's item here? So If it's an enum Or it's a match I assume match is a thing here No, it's not is it fine Okay, fine, uh If tie is that If tie is that then we return the enum otherwise Matches a statement or expression not an item We'll get to it in later exercises. I think for now. We'll just do it the way the way they want Um I also want this to not have to do this I want to operate on proc macro to Actually here proc macro to token stream Actually, what am I doing? Let's just do this instead Sorted impo args input Then because now we can just do this Because now this can return quote Um, let's spend I guess And here I want Tie dot span To Compiler error, uh, and it should say Expected enum or match expression. Actually, why am I even doing that? Could just do this use quote and Expand proc macro create types Oh, this is a private function. So that's fine Um, it does a return doesn't it? So we'll do this this will just take input Which is going to be a sin item It's going to be tie really think that would be necessary. Yeah, something's not right there I guess this is really going to be Wait, why is there not a span on sin item? That seems not quite right. What happens if I just do this? Then it won't highlight the right thing Like if I if I now go here and take this thing and go to main And then run this what's it going to tell me? I mean that this is all fine Macros is going to items must be eliminated with braces or followed by semicolon macros that expand to items That seems unfortunate. Oh, actually that basically generates the right thing I'd prefer it if it highlighted the next item Um There is Um No, there's not I I just tried it See Use sin span span. This gives me a sin item input span Run that Oh, sir span is private. Oh, spend Spend I can't spell can I Great. Oh interesting. This is all sorts of not what I wanted Also, let's stop doing this debug print Macros that expand items must be delimited with braces or followed by semicolon Oh, I see this arguably should still be input probably really does not do the right thing Oh, you're right. I'm being stupid Because we're replacing the whole thing and so it needs to yeah, you're totally right great It's still highlighting something kind of weird It really ought to highlight the whole item But it highlights only the Visibility which is kind of weird. See so it actually wants us to not highlight that I mean, I guess that's fine Like we can easily just do that It seems worse, but okay All right, uh, so progress so three Out of order at this point we have an enum. Let's do this Oh, where this is now we're getting there right now. We have an enum. That's out of order Uh, when your implementation notices a variant that compares Lexiographically less than the one of the earlier variants You'll want to construct a sin error that gets converted to token stream By the already existing code that handle this conversion during the previous test case The span of your error which determines when the compiler places a resulting error message will be the span of whichever Variant name that is not in the right place Ideally the error message should also identify which other variant the user needs to move this one to be in front of Okay, so Here what we're going to do is If it's one of these right, so I think the suggestion here is basically have Another function that returns a result and then down here we're going to Turn that result into a compiler error if necessary Right, so if we look at 02 Sin error provides a method to render your error as a token stream Great So up here Um, the recommendation here is basically have the oops have this return a result I guess proc macro to token stream Or a sin error And then this can now return a Uh, what is this? How does sin error work again? Sin error new great Sin error new Oh, I don't know what span to give it though Yeah, see It's also kind of annoying Um, the problem here is parse macro input produces A token stream so I can't put it in here because it wouldn't have the right return type And I don't want to use the span of the input because I want to use the span Of args basically See it's unclear what span I even want to use here I don't think I want to give it a span So I think actually in this case The span of args is call site What is call site? Oh, I see it's the call site of the macro I see so how do I get one of those? Ah, I see I see I see I see I see so I hear Really what I want to do is new where I give proc macro to span call site And then the error message I give is that And then this is going to be Okay, this This is going to be unwrapped or else e dot to compile error which we can write as sin error to compile error But we can't because the rough And it's great So now let's see that it still produces the right thing. Okay, great. So for three Now we don't actually check that they're out of order great So we're now to the point where we need to check that the variants are in the right order Now check variant order So this brings us back to we have a Sin sin item Which contains an enum. So we have an item enum and we want to walk the variants Right, so for variant variant and variants and e dot variants probably by rough because punctuated implements Itter right. Yeah, great So it's going to give us a bunch of variants and each variant has an ident So Name is going to be variant dot ident Dot true string And now what we want to do is check that these are in order. One thing we could do is just Allocate Basically parse all of them into a vector sort the vector and see that it is the same as the original vector The other alternative is that mute If name Is less than last Then we return Then we want to return an error because in this case um Name should have been before last, but it was not So in this case it would be Variant dot span So we're going to have to bring back use In spend And the error message is going to be variant out of order And then last is going to be name See how that works Variant out of order something failed. Is that what we expected? Yeah Now it also says ideally you would tell us Where it really should have been, right? I mean, let's let's first see whether it gives us Okay, what does it expect us to give? Something failed should sort before that failed Ah, I see I mean, this is luckily pretty straightforward Names is going to be a veck Uh if names if not names is empty and names Names dot last dot map l is Uh If the if Name is smaller than last On our false Eames push name So in this case we now have all the names in order So we know that names is always going to be In lexiographic order, right? So what we can do is Uh Should be Is going to be uh names dot binary search Pretty sure there's a binary search in this summer. Uh, nice theme. Do you have a setup for download? Yeah, I have a there's a separate video where I go through how my whole setup works You can take a look at that if you want Uh, so basically what we do is we're going to search for where this name should have been in the list Uh unwrap or else I guess we can match on that Um Okay means that the name was already in the list elsewhere. So I think we can actually do Which the compiler would already complain about So we can here do unwrap error So the error type here is What index it would have come right before right after? Asserted while maintaining. Yeah, so this is where it would have been inserted Uh Which means that The element at that index is the element. It should have been placed before Right So Here Here what we want is format Uh What was the merit message this should sort before that Where that is a name should sort before Names should be so this would be I guess what should we call it should be is a bad name for this variable. It's like Next Lex I See how that expands Oh Something failed should sort before that failed That seems promising Perfect All right progress oh four Uh This doesn't similar to the previous we want to ensure the macro correctly generates an error when the input genome is out of order This time it is using an enum that also has data associated with each variant Okay, so in theory that should be a freebie Oh, it's not Huh, this seems wrong Here there are a bunch of warnings Oh No, this is probably still us Probably doing something wrong So if we expand this what does it expand to? Oh What? Oh, it doesn't emit the enum That seems like a problem Uh, I see what's going on here. So So We don't emit the enum if we give the error But we should right we should still keep the original enum in there Uh And because we don't emit the enum a bunch of other warnings appear So because we know that sorted is never going to change the enum. I think what we're going to do here is boot out it's going to be that out dot Uh So this result Is really going to be empty. It doesn't have to produce anything and then we're going to do I think out is just going to be equal to input and then if let error e Is that yeah? Yeah, this is what we're going to end up doing right then Uh out dot extend this Cannot infer type for a This should be token stream yes token stream from Great and now this okay doesn't actually have to return anything because we're always going to emit the enum back out This input is we can just clone here. That's fine Oh This is going to clone that input stream Great. So now we still emit the output, but also the error Great. All right progress Uh five match expression get ready for a challenging step Much bigger change to the other so far great Not only do I want sorted to assert the variants of an enum are written in the order of enum definition But also inside match expressions to match on that enum Um currently their procedural macros on expressions are not allowed by the saleable compiler to work around this limitation until the feature Stabilizes will be implementing a new Sorted check macro which a user will need to place on whatever function contains such a match Mmm The sorted check macro will expand by looking inside the function to find any match expressions carrying a sorted attribute Checking the order of the arms in that match expression And then stripping away the inner sorted attribute to prevent the stable compiler from refusing to compile Know that unlike what we have seen in the previous test cases stripping away the inner sorted attribute will require The new macro to mutate the input syntax tree rather than than inserting it unchanged in the output token stream as before New procedural attribute macro called check Parts the input as a sin item fn traverse the function body looking for match expressions Easiest if you use the visit mute trait from sin Yeah, see I wish the visit mute or visit were documented because currently they're a bit of a pain For each match expression figure out whether that's sorted It should be that we get these present. Okay Great, so Let's copy this into our main again So we have a thing to look at And back resource lib All right, so we're going to define up here a proc macro What did he say? Right Yeah, so procedural macro be the same thing. We're going to parse that as an item fn and let's do Let's say that sorted impulse is a bad name to say sorted variants And then down here we're going to set say sorted match Great, so this is still going to include the original thing that was Given with the attribute So now we're going to have to write Sorted match And the input is going to be a sin item fn Right because it's an annotation on a function and like before We're going to reuse this Error type of ours Great, um, I guess we do get to use visit after all Vis no, it's not what I want back to sin And we want visit mute Okay So here we're going to have We're going to have a struct lexio graphic matching And we're going to implement Sin visit mute Visit mute for that type And in particular In fact, I think this called out which one we wanted Visit expert match So we're going to want That function Which is going to take That's not what I wanted Visit mute This function This is going to be a sin expert match And specifically remember how Down here Oh, that's right. We can't we can't do the same thing. I lied If we success oh man We have to remove the sorted attribute regardless Because otherwise the the program won't compile in the first place So I think what we want to do here is be a little bit more careful um, and specifically we're going to do that by Not having this business instead doing this so Not going to do this thing Not going to do this thing and instead just do Okay, so how is visit mute supposed to be used? I guess it's supposed to be used like Mute F Right, and then we're going to do F F dot visit item fn mute No, it's not true. We're going to do lexicographic matching Dot visit item fn mute Yeah At mute f And now we still have the f and at this point what we want to do is um The this lexicographic walk is going to insert the compiler error for us as well So all we really want to do here is just return back out the F might not quite work that way, but you know roughly um Great. So here, uh, what we want to do is No, it should be m for match um Okay, so what we now have is a soon expert match So the first thing we have to do is check that there's an attribute there that called sorted So if m Adders um It or any a A is equal to sorted If not that, uh, then return. Oh, we also need to add The visit dependency Really? Is there a visit mute as well feature? Apparently Okay, so visit mute And I did something silly, didn't I? um So for the attribute Oh, I'm gonna have to parse it as well, aren't I? um Who is going to be annoying? Because what I really want to do here is like adders is going to be Ah, no, that's fine any A dot path Sort of equal to sorted inside the current one Oh, you're right. I can't even do that. I needed to visit all the arms as well Okay, fine. So check All right, and down here what we want to do is we want to walk all of the arms although there's a Actually, we can do a little bit better because visit mute has this thing Which I I assume these functions are the default implementations um Where's visit expert? Yeah So here we can just do soon visit mute this with the visitor Which is self and Keep keep Recursing so really what I want to do here is if this is true then Check the variance But crucially this is not quite sufficient what I want is Well, sort of I guess I want to remove sorted Right, so we need to make sure that we remove this attribute which we can do with retain sorted I probably can't use equal to string but it's easy to fix later Uh, and then we want to walk all of the variants of the expert expert match Okay, so for each I guess each arm names Is going to be a vac new for arm in Arms iter We're basically going to pull the same trick here as we did above Oh It can also be multiple patterns, which is all sorts of weird with this I Like what do we even want like if the user writes something like match foo And they I guess they annotated with sorted and then they do a or z this b This Z this I guess I mean like is this a problem. I don't know. What if they do z or a this What are we going to require? Right, this is why every arm has multiple patterns I think we just want to look at the first pattern At least for the time being So name here is going to be arm pats iter Next we could stringify the full pattern. I suppose it's a good point Um, is that what we want to do? Can I even do that? Like does this implement display? It's not clear to me that it does I mean an implement is too token I suppose Yeah, like I could I could quote it and make it a string Um It's also unclear that we're going to require the full pattern to be alphabetic I'm not I'm not convinced. That's the case I think we're just going to leave it like this for now. Oh, it gets even worse Uh Like what if the user writes this? Right And then they write like x at a whatever a at z Is this wrong or not? Interesting No only except items won't work either, right? You still run into the same issue like You want to handle things like pattern structs tuple structs I think we do require anything that's named. So I think the options are like tuple structs structs and idents And for the ident we only care about the sub pattern I think really what we want here is a Get arm name which takes a sin Uh Pat And gives us a string It's going to match on arm It's going to match on the arm And if the arm is sin pat Struct Then it's going to do something if it's tuple struct Tuple struct And if it is an ident So if it's an ident A pat ident then we're just going to match into the sub pattern. I think So then we're going to do get arm name of I guess option string Then this is going to be i dot sub pat dot map Actually as ref map sp Actually, I don't care about the at But I do care about this That gives me a sub pattern Which is going to give me So this is going to be an end then the other thing that's going to be annoying here is to get out the um Is possibly at least to get out the spin, but let's stick with this for now So this is going to give me The sp anything else is none If it's a struct Then I want it to give me it's probably just like the path Yeah Then it's s dot path Uh, which probably has a two string It does not why does it not have a two string? Let's try it and see if it does I feel like it should but i'm not sure it does Um, and the tuple start is probably exactly the same Where it has a path. Yeah Actually, you're right. This could be sin Pat Ident I could just like match into this Where what I care about is sub pat Is some And then I don't care about this, but I do care about the sub pattern Um, I do think I care about other things than just idents like for for something like Matching on the struct like that seems totally reasonable to also order if it's an ident the question becomes do I match on the The bound variable name or do I match on the variant and I think matching on the variant is actually the right thing to do um Because the named binding is just going to be a variable name for example the named binding could be nothing Yeah, exactly. The sub pattern is if I write at least if I understood this correctly like If I do foo at bar zoo, right? It's the bar zoo I want to sort by not the foo because the bar sue are the variants of the enum And so that's the thing I want to sort by not this and this is the sub pattern um The other question of course is whether Yeah, so path doesn't implement two string, which is a little awkward Dot arms, that's fine Yeah, so the real all of these come down to I have a path font I want a string representation of that path which is like maybe what that gives me but Imple path so unhelpful. What does path segment give me? Oh because it can contain types and stuff too Um It might actually be that what I really want here is uh Is to I'm surprised the path isn't display That seems wrong right because I guess it implements two tokens so um So can I do like this? Or Better yet, what does two tokens give me gives me a token streams In fact, the other option here is to do this But that's not gonna be very nice. I actually want to print the path itself Um, yeah, I'm gonna need quote aren't I? quote quote Okay, so here We're gonna do just the sort of silly This ts into That's fine Um, and of course, yes, I do need to include this Path doesn't implement display Pattern doesn't mention fields. That's fine Okay, so all of these now are about how do I get a string for a path? um Okay as string So I know that doesn't work, but let's at least Move all the code to use this. So this would be path as string s dot path This would be the same This would be Path as and same here Definitely less efficient than it could be Um, and now the real question just should just be this Uh, oh, I guess This is going to have the same property as up here where This where this is going to be Get arm name for now, although that won't actually be an unwrap Down the line variant dot span. So this is actually going to be arm dot span 59 This Yeah, we're gonna have to figure out how to Change that too I think this is so this is where we're going to inject the error if something is not sorted And I think we need to inject that Into the body of the arm. Maybe it might not matter. It might be we can inject it anywhere There is a path dot is ident Mmm Okay, so that saves us from this one But it doesn't really help with the others where I want the full representation of that path So here, I guess what we're gonna have to do is something like m dot Where's the thing where I get? No expert match Um on a given arm body box expression It's going to be awkward actually to modify this Oh, I guess actually what I can do here is just like Uh an error is going to be a Vec of sin error So here I can just do errors dot push this So we just accumulate all the errors that we find um And then at the end here, what we'll do is Uh Do that and then T s dot extend Um, I guess we can derive default for this Selects your graphic matching default lm that extend lm errors Into iter Map For each error we need to do To compiler error So now I think the only thing is uh 49. I think the only thing now is the whole Path to string business No field errors Yeah I mean I It might be that we can just do this Format should sort before io That seems promising Did that really just like work that hard to believe? Oh, hey, finally It's funny how I think most of the people who watch these videos have not watched it live The advantage is now you can ask all the questions. It's the real upside to watching it live Hey, great that worked Ok