Maybe not somebody know Sorry, but the clicker doesn't work is it oh, okay Okay, sorry about that Okay, thanks for the introduction here So why the previous talk was more about improving actual lettuce reduction what we are targeting more is the analysis of lettuce reduction, so more like getting more into the How can we predict what an adversary can get out of lettuce reduction and? before stating our results, I want to take a step back and Say a few words about what we actually require from algorithms for crypt analysis in this case in particular for for lettuce algorithms and and I would argue there's a Essentially three requirements one is obviously you want the reduction algorithms to be practically performant This allows us to play around with them in smaller to moderate dimension and see how they behave We also want them to be asymptotically performant because that Makes sure that nothing weird happens in larger dimensions where we're not quite sure where we can't run the algorithm anymore but These two requirements are pretty standard for any algorithm essentially, but there's a unique requirement We have in crypt analysis, which is saying that we want as an average case prediction Which is as simple as possible so that somebody who's designing a cryptosystem can apply it easily and Yeah, I can sort of figure out What an advert how long an adversary needs to run an algorithm in order to break his cryptosystem and? One might think that the first to actually imply the last one But this is unfortunately not true, but you can use the first two properties of an algorithm to guide you through to an average case prediction So let's see what we've got in current algorithms. Well, we obviously have BKZ last talk was about it BKZ and BKZ is widely deployed and use a lot mostly because of its practical performance. It performs really well in practice You run it and yeah, it's it's really nice on the other end It's asymptotic performance is actually pretty good, too. So by now there has been a lot of work in it in the last few years and We know we can do a little better in the asymptotic worst case But it's roughly in the same ballpark. So that is not too much of an issue However, it's average case prediction is still it's still quite notorious. It's The state of the art right now is to run the simulator Where you generate some some typical input to it then you run the simulator and then you look what comes out The simulator is based on some really well-founded heuristics but also on some heuristics, but it's not quite clear if these actually hold and it is quite inconvenient to run for to apply for For crypt analysis or for somebody who just wants to set the parameters of this crypto system And and so we are trying to improve on the situation. What else do we have? We have on the other end we have the slide reduction algorithm That is not as widely known as BKZ and not as widely used and that is mostly because it's practical performance is believed to be quite bad on The other hand it's asymptotic perform performance is as good as it gets at this point for lettuce reduction So it is asymptotically the best algorithm we have and also it's It's asymptotic analysis is actually really clean. It's a clean analysis. It's a nice generalization of the LLL analysis and Out of this clean analysis actually also a very simple average case prediction falls just out of the bottom so Unfortunately, though Yeah, it is not used a lot because it's practical performance. It's not very good And so natural question for us was obviously can we get the best of both worlds? And our answer is the self-dual version of the BKZ algorithm We want to our version of the algorithm to be performant as well or just as performant as because a so we might model it after the BKZ algorithm and Using experimental evaluation. We can show that it is just as performant as or comparable to BKZ But then we sort of like try to sidestep the issues that BKZ has with this asymptotic analysis and its average case prediction and so Using these sidesteps and using maybe slack borrowing a little bit of the techniques from slight reduction We actually get an algorithm that is asymptotically just as good as slight reduction and we have a very simple average case predictions meaning that an adversary can just And kryptonites can just Use a closed formula to predict what comes out of this algorithm. So You don't have to run a simulator anymore You can just yeah ask for a block size and we can say what roughly was gonna come out of so, why do I say that Why do I say that slight reduction behaves so badly well It's only based on one experimental study that has been done so far that included slight reduction And that is one by a garment Gwen in 2008 who actually also designed slight reduction and here so on the x-axis you see the block size parameter of the Reduction algorithm, which is sort of like a measure of the time spent on it. So of the runtime and here you see a normalized measure of the of the shortest vector found by the algorithm and So the law is better and as you can see BKZ clearly outperform slight reduction In this in their experiments and that is why no one has used slight reductions of the since then anymore And it is sort of like it was more considered as a theoretical algorithm but since we Were able so we borrow techniques from slight reduction And we also have to implement our algorithm and then we had all the tools ready to also implement slight reduction relatively easily So we included it in our experimental study and for their experimental study Here's what we found the x-axis is the same again block size a measure of the runtime of the algorithm horizontal Vertical axis is a room hermite factor against the normalized measure of the shortest vector found and yes And rather small block sizes up to around 50 Which was the limit of the previous study as I just showed you So BKZ and do BKZ clearly outperform slight reduction, but as you increase the block size actually Slide reduction becomes quite competitive as well And as our study shows our dual BKZ algorithm performs comparable to to BKZ So that's on the that's our result on the experimental site We also have another technical contribution that should be of or we hope to be of independent interest For this. Oh, yeah, one more thing All our code and data is available online. So if you want to run your own analysis on the data or want to add more Experiments or you want to play around with the code? Please feel free So and then as a at the third contribution recall that What BKZ does over and over it looks at subject at projected sub lattices of the Of the of the lattice or of the lattice basis and it applies this one operation over and over right it applies this This SCP reduction where we take the first The first vector of a lattice basis and we make it as short as possible And because the sum over these is a it's a lattice invariant all the other ones get get larger And so this is how this mess is shaft shifted from the left to the right and it tries to make this the shape of Of a bit of a basis as horizontal as possible By just running this over and over I go a bit more into detail later But for now, we actually have another tool in the toolbox for lattice reduction And that is you that is known as dual SCP reduction and what you do there instead of Minimizing the first vector in this in this shape what you do is you try to maximize the last one It is not quite obvious how to do that. But what you do is you transition Into the Into the dual. Well, yeah a few words. Yeah If you maximize the last one again the sum is lettuce constant stays the same And so the other ones get shorter and again you're moving Mass to the to the right and you're trying to get this this whole thing more horizontal And how you how you achieve that is you transition to the dual Of of a lattice and you compute the shortest vector there and you insert it in the dual and then you go back into the primal But unfortunately computing the dual of a of a lattice basis is actually It's asymptotically dominated by the svp oracle step But in practice it is it involves matrix inversion So it involves the factor of at least n2 to 3 which Yeah, it's kind of annoying, which is also maybe a reason why slight reduction wasn't implemented before It uses dual svp reaction But what we come up with is an algorithm that actually allows you to Compute this reduction of a of a basis without ever computing the dual of a basis. You never have to do this This dual step and or this this dual computation And the dual enumeration so you're implicitly running an enumeration in the dual without ever computing the dual And if you look at the algorithm, I don't want you to understand this right now I just want to show you this is a This is a dual enumeration. This is a prime enumeration as you would Like yeah as you would implement it And they're structurally just really really similar. This is all I want to show with this with this picture As you have a few times you you run this loop in a different order Um, you have sometimes you have a binders where you have a plus here You divide here where you multiply here But essentially structurally they're both the same the same and it turns out they're also as you would expect from such Algorithms, they're also just as efficient. So the This is the rate of enumeration in several dimensions Um, so the higher the better, but the rate of enumeration this is in times 10 to the seven nodes per second It's essentially the same in the dual and the primal and also Adding this to like if you have an implementation of an of a primary enumeration adding a dual enumeration It's actually not very hard because you can just adapt your implementation really easily And that's what we did in order to implement it We added it to fpl l l and this part is actually No part of the main branch of fpl l l. We are also working on it to get it Get our reduction algorithms included, but this is already part of it if you want to play with one with it feel free all right And this is our results now. I'm going to start going a little bit more into To the details and first cover some preliminary that Yoshinori didn't cover For ones, we are going to have to talk about the dual lettuce a little bit The dual lettuce is defined by all the set of Vectors in the span of the lettuce such that the scalar products with any lettuce vector is integer And there's also a notion of a of a dual basis. So the first one is It's a as a relationship between lettuces now We also have a relationship between bases. So as you know lettuce can have infinitely many bases but for any basis if you Compute the basis d such that they have the same span and they are sort of like the inverse of each other in some sense This means That d actually generates the dual of b. So they're actually generating dual lettuces And Uh, we also know something about their the relationship between the grime schmidt authorizations we called bi stars as a So orthogonalize vectors the length of the orthogonalize vectors And their relationship to the duals is also given by by this formula So their length is one over the length of di star or star di To the minus one Why do I write star di well it turns out the right way to look at The authorization of the dual is to Authorize it from right to left rather from then from left to right. So you start from the last Last vector you authorize everything before that And then you just keep going and then if you do it that way then the gso are nicely related as well and in particular we have That b and star if you substitute an i for n for i then b and star is just the length of the Last vector in the in the dual basis and because we are authorizing the dual basis from right to left This is just a This is just a dual vector No authorization whatsoever. And so in order to maximize this We minimize this dn and so this is where just going into the dual comes from and svp reducing the dual This allows you to maximize the b and star. This is where it just comes from And so a couple of more things we need we're almost done with the plenaries, but we need a few more equations and in particular we want to Talk about some bounds that we have on scp that are that are useful in the analysis Obviously, we have a minkowski's bound which tells us that for any lattice The shortest vector lambda one is the first minimum It's the short it's the length of the shortest non-zero vector in the lattice And this is always smaller than or of square root n times the root determinant of the lattice And that is independent of the basis by the way. That's also interesting, but If you take if you replace this Lambda one with the actual with the actual vector that you have in the in the first Yeah, the first vector you have in the gs or after you svp reduce Your basis and you take the log of this equation. This is the equation that you get out of this also the Determinant is just It's just the log of the determinant is just a sum over the shape of the basis or the log of the other bi stars So you get this if you take the log of this equation, you get this a fine inequality that you can exploit as we'll later see in Where we're going to model these these log bi stars as As variables and so the output Or like yeah, this output variable is going to be an affine combination of input variables Furthermore, we have the gaussian heuristic which looks very similar It actually in this notation it pretty much looks the same the gaussian heuristic says well for random lattices So there's like there's an ocean of random lattices that where you can prove this heuristic actually rigorously, too But in this case, uh, the first minimum is actually quite close to order of square root n times the root determinant of the lattice Um But for constants that we actually like we can compute these constants and So we actually get a some something that's somewhat stronger because we get a rough Equality here rather than an inequality and we can do the same trick where we take the log and we get an Yeah, we get a rough equality again an affine one However, I put a star there for a reason And it has been shown that there's a caveat with their previous experimental studies I showed that in the context of block reduction um This uh, this gaussian heuristic is only accurate if you increase the block size large enough or for large enough blocks So only when the block size grows beyond 50 or something. Uh, that's when the gaussian heuristic becomes accurate Below that it's actually horribly inaccurate And that is actually where a lot of the problems for bkz stem from so as we will see now Let's have a brief look again at bkz bkz As I mentioned before it takes this this svp Um operation where it tries to minimize the first vector and it applies this To the first to the first block of the basis for the first sub lattice The block size parameter is better um And it then just keeps doing that all the way to the left until It reaches the end And when it reaches the end we are kind of stuck right the the windows at the at the right We can't move any further, but what bkz does is it just simply makes the block smaller and smaller and so at some point um Yeah, so and when we are when we're done when the block size has size one Well, then the problem is solved trivially and then we start over this was a tour of bkz as uh, yoshinori already already mentioned um And uh bkz just does it over and over again. And so how would you analyze this well the state of the art analysis? um Looks again at uh, well, what happens What happens in this step? Well Let's say we we model all these these bars These are our log bi stars. Let's model these as uh variables and let's say we know what's what's coming into the algorithm Then using minkowski's Minkowski's theorem. So there's this inequality here we can we can write the output as in a fine equation of of the input And in the next step we can do the same thing again And we do that over and over again. And what you can do is what you can show Is that you can set up this um this dynamical system and then You get for if you have upper bounds for the input then you get an upper bound for the output if you apply this Dynamical system once and now you can use the techniques from the dynamic dynamical system analysis um and prove And look at the convergence of the system as it's a fixed point the fixed point is going to give you Uh an upper bound on what comes out of this algorithm and the convergence is going to give you an upper bound on the runtime And this was done by aran So you're in stile. That was a very nice paper. That was applied to bkz and now the hope is that if you replace the minkowski's bound by the gaussian heuristic that you would get a very easy um average case analysis out of it, but unfortunately this is not true because remember We have these uh smaller the smaller blocks in the end and here the gaussian heuristic doesn't hold and so unfortunately Just can't you can't just take the the analysis in this paper translated into a I'm translated into an um average case heuristic analysis But this is essentially what the bkz simulator does by the way It is trying to do that and in the end it has to resort to some sort of unproven assumptions About what's the shape of of the tail is going to look like And this is where a lot of the problems for bkz come from in the average case prediction And this is why you need a simulator and can't just use math and a closed formula and so How does this uh, how does it differ to to self-dual bkz? Well, we said we want to retain its um Okay, we want to retain its practical performance and so What we do we just start out the same way as bkz as we take the first block We um as a period uses and we shift to the right But once we get to the end We have this last block and there's uh what bkz made smaller and smaller block sizes and reduce them And that's where all the trouble comes from we say well, we don't do that Instead in the end we apply a dual as a period action We leave the block size that is but we apply the dual as a period action and now we just move the uh This doesn't work We just move the block in the other direction back And then once we get to the other side again, um We'll start doing svp reduction again. And so you have this window of size better. That just moves back and forth on this basis um and um We mainly do that because we want to get rid of this small tail or of this tail of bkz And you might think well now this looks more complicated than bkz So the analysis uh might be more complicated, but as it turns out You can view this as also doing one tour all the time Just you do a bkz tour and then instead of doing another one what you do first is you Uh compute the dual of the basis compute its reverse reverse all the variables Uh and then you start over so what you're doing is you're running bkz Go into the dual reverse the basis and start over to bkz again reverse computer dual et cetera And so this actually is a much simpler system than uh, what the bkz system looks like um and because we don't have smaller blocks we can actually simply plug in the gaussian heuristic and um and get an average case analysis out of this for free essentially and One more thing that's very interesting about this is um It came up in the previous talk that often about lettuce reduction There's an additional assumption being made that the output Of um lettuce reduction Follows a straight line which is known as a gram schmidt No, the geometric series assumption, sorry So schnoz gsa says that the output like this would actually look like a straight line at the end of the lettuce reaction Or are roughly like a straight line well if you look at the fixed point of of our system and And it actually Proves that this is true for under the gaussian heuristic. So for our algorithm actually the gaussian heuristic implies the gsa Module is a little window at the end. So for all the other For all the other vectors the gsa actually holds true if the gaussian heuristic is true And so this is this is not known. This was not known about bkz or not known about bkz But for a third dual because we don't have to do an additional assumption um I was going to talk a little bit about the dual enumeration, but uh, I'm out of time and I refer you to the paper for them Thank you very much Any questions? Hi, thank you for the talk. Uh, just a quick question when you compared, uh, the performance of bkz to 12 bkz Was it the full bkz or the truncated one? Uh, it was truncated So, yeah, we went up to block size 80 at like at some point you can't Uh, run it. Uh, it's a full bkz. You have to do an early termination. That was a question right there And so that's what we did, but that's also what we did for dual bkz So the the terminating condition in in the self dual bkz It's also a little tricky because you're always changing the basis because you're always Computing the dual and so, uh, you have to be a little bit careful with that, but um, Yeah, and I mean like the dynamic systems analysis shows that, um After a few rounds, you don't have to run it anymore. You you can terminate Also questions, okay, let's thank the speaker again


I told you guys! I told you! I told you! I told you! I told you! I told you! Ni wanbia, we are not playing today. We did not come to play. And of course, these are all about hip-hop. I love just all about hip-hop. And throwbacks. Yes. Throwbacks in hip-hop. Throwbacks in hip-hop. Of course. You hang out with me. Your favorite presenter is Lucky Idol. My name is P.H.Y.L.I. in E.J.E.A. And feeling Jinn hang out with... The big doctor, DJ Epic, can you epic the DJ? Uki pandam nizam wicha dakitare wa. Wa masanturi na konbia. Lo tu kona omusani. Tu kona omusani? Yeah. Aha, ame pandan begge. Hey. Anyway, we have a singer, rapper, songwriter, performing and recording artist all the way from Nigeria. And he goes by Lolli B. Can you give that for Lolli B one time? That's what's up. Yeah, how you doing? How you doing? I'm doing. Okay, good to have you. I'm Kav, still. Oh, you're still Kav. Yeah, but I'm good, I'm good. Right. You're from Nigeria, but you're currently in Kenya. How long have you been in Kenya? And because I've seen best of you in music, I've had you speak a little bit of Swahili here and there, so... It doesn't sound like a newcomer. Yeah. No, really. It's because I used to have a lot of Kenyan friends in Malaysia when we were studying in Malaysia. That's why I cry. So, that's why you know a little bit of Swahili here. Yeah. So, I love that. All right, one thing my Niger friends complain about when they come to Tufa for is the food. They're like, you guys don't know how to make good your love. I do not know. How do you have the same experience or how you love in Kenyan food, how far? For me, I love experiences, new experiences, so I love to explore. You can try to make your culture with someone else's culture. It is a culture for a reason. I'm telling you. But therefore I had a lot to say about Kenyans, a lot. No, me and my fan of chapati. Chapati has always been my favorite meal. So, do you have chapati in Nigeria? No. No, right? Yeah, I see they find it so it's different. And you guys have Fufu and our Ugaris a bit are tougher compared to the Ugaris. That's why I keep having problems. The Fufu and Ugaris, you guys are getting it twisted. The only thing in Nigeria you can compare to Ugari is gari. Yeah. We call it eba. That's the only thing you can call it because it's strong like that. Yeah, but Ugari is too strong. And we don't eat ours, we egg sauce. What do you eat? You as well. We soup.  Yeah, we have different types.  And we never do that. You do it here, oh my goodness. Fish nanyama takulamot. It's trouble, you know, but you make it work. Seriously, she take out like this December I did a cookout with Jeff Alejandro. Yeah, yeah. So I made noodles with chicken and omelet by the side. And then I made another noodles with beef for another noodles. It was awesome. Okay. We did not see that. But you also love to go. What's your favourite meal to make? For me, it depends. But I always love anything that has high protein.  On yama choma? You even mentioned it in your song. That was so slick. Yama choma and that was like. This is a tag from Nairobi to Kenya too? What's that tag line? Naijia to Kenya. East side Nairobi. Chilling with Kenya and jail. Yeah, you hear that, that's amazing. I will forever rep omodja east side. I told you. I told you. I'm telling you. So on this show, it's just that these are, they are still a bit fresh, but we usually have a segment on this show, on a day like this every past day, we call it our spito. So we usually have two rappers come through. So it's always east side but west side. I'm always repping east side. East team west side. So now that you said omodja. Omodja Peter, she took me as family. So it's been a blessing working with Milashie. And he's part of your EP, which we'll talk about in a few. But first let's take it back a little bit. Let's talk about your background in music. When did all this begin? Because you do drill and at the same time you also incorporate a little bit of Afro bits, Afrofusion right there. So tell us about that. I've been writing songs for some artists in Malaysia. Sometimes I write for record labels and producers. So I've always been camera shy and all that. So I was like, you know what? Last day I'm just going to try to do my own thing and bring it out and show people that I can actually serve them a buffet. So what I serve them my EP is a buffet. A different sound, different genre. To tell them I can do it all. Alright, I love that. Another thing you spoke about is the fact that your music inspired by your own challenges, the ones you've experienced and pain you spoke about that. But I don't know if you're comfortable enough to talk about that, but maybe just in a few words, what are some of those challenges you spoke about? At the point in time, you know, being in Nigeria comes with a whole lot of difficulties. You get stereotype instantly and you get to do with the stereotype. So the only way I could deal with the whole stereotype and then when you go to Malaysia now you deal with racism and stereotype at the same time. But at the same time, for someone like me, I always find a way to voice it out whether through music or acting. So I don't keep it down.  I'm cool with it. For me I'm comfortable, I'm used to it now. So it doesn't face me however you feel about me or my country. But regardless, my country is still doing their thing like musically, boxing wise, UFC. So there are still good ones trying to do good things out there. So the hatred is just people just jealousy. You think so? Because everyone is always like I've got power in Nigeria. Nigerians are bad news.  That's the same thing they say in Malaysia and yet, all the Hokiaw Sturros after the Nigerian boys here in Kenya, most Kenyan men are not trying to act like Nigerians so they will be accepted. So that women will like them. But most of the time we're just being us.  That's how it is. We don't care but we do our things. Right. Now, because of time, I want to talk about your EP. It just came out and your EP is sort of Villain. Who's the Villain? Who's the Villain in this story? Are you the Villain of your own story? Who's the Villain? Are they the fans, the critics? Who's the Villain? Nah, for me I have a song there called The Villain is everybody, I was speaking for myself or I feel like everybody can relate whereby you're that good guy you keep doing good and then people keep making you feel shitty for doing good and then you're like you know what, I'm no longer the victim I'm the Villain. You are now the Villain. You crossed over. Like any energy you give me I'll give it to you back. Right. So you reciprocate. If someone gives you good energy you'll give 100% If it's bad you're going to Yeah, I'm just going to ignore it. I hear that. Right, another thing I want you to talk about is the artwork. There's a song called Weird of the EP but before we talk about the artwork was really creepy though can you talk to us about that? I don't think we have it but it's on the song. You can call it you can call your EP Weird if the art doesn't match it. You just say the EP? Yes I did. You saw the artwork I'm talking about? Yes, definitely. This is my guy. I've seen everything. That was a 3D art only because I like creative artworks like Ellen's and so I decided to try and put it there and put my girls in it and see how it looks and because like I said I'm being considered weird because of where I came from and travelling around the world gave me a whole wide exposure so I feel like me being that weird put me like 10,000 steps ahead of a lot of people so that's why I said let me accept what I'm being called I'm weird. You're weird so you accept being weird you accept being weird and he loves aliens? Yes Do you know aliens what the other day?  You can see in Miami they were like over almost 100 almost 100 I've never seen so many but I've never they say it's just people fighting 4 people fighting they were like and guys who were there they said it was in a mall people coming from a mall they said they saw aliens they were like giants they were like 10 feet tall beings but there were a couple of them but then would you take a video? Yes I want to take a video I want to take a video I need an evidence Yes there is no way but the video they showed it was not clear it was just like like usual but then it actually makes sense because they were like almost 100 vehicles in those cop it's not like one cop let's say like the four of them and they are all armed and there was a there was a patrol the police also had the chopper there was also in the air so why would they bring a chopper for people who are fighting and then they cut off electricity 30% of Miami did not even have electricity guys were like those were aliens they need to come clear people who saw them they need to come clear yeah they need to come clear the people who saw them they said they didn't have feet though they were like those things were just like moving but without feet it was weird and he said he likes aliens so yeah people are here I would just anyway anyway that is off topic if you didn't know the story you could just go search about it I think everybody has done a story about it right about now and speaking of your EP the one that just came out you have you have featured some Kenyan artists so let's talk about that how did you guys link up first of all I came to Kenya around January 8 so I was in that productive linking up with people or socializing so when I went back to Nigeria and I was like Kenyan Jewel artist and then what I saw was Milashie's music Cha Cha Slide featuring Mr. Right from Brooklyn Boys so I texted the cinematographer for it and he linked me up with Milashie and we clicked we became more like family and then we worked on the song and then my heart became slim slim yeah slim someone just brought him to my house they were like yo Loli you need to do a song with this guy he's good his name is slim me I was like I don't want to work with him but I just want to stay on my own and then when the nigga started rapping I was like he's got it he's good but I feel like he can do way more singing so I told him bro I sing he said yes and then I played the instrumental he sang you played the instrumental so you play okay wow let's finish I like you you can sing let's hit the studio and that's how it made it happen and that's why the song is the last song oh nice so how many instruments do you play now that you've mentioned that how many musical instruments do you play I only play one piano I guess just piano right I'm so good at guessing someone will sing I guess piano let's talk about the EP what is the whole story influenced by like we said it's a little bit of everything here and there but mainly what is the theme of the EP is let's just say the EP is just a way of me being petty I just summarized my life because my whole life is being like I'm being bullied for most of my life and whereby relationship, family so I always try to touch the relationship part and then family in general you know like you Kenyans when you guys come back from abroad your family will treat you nice now everybody expect a whole lot of things from you but for us Nigerians when you go abroad whether you go to study a friend of mine who we did high school he texted me he said Loli I hope you're not going to come back with degree certificate saying oh I went to school I'm like what do you mean bro your mates are coming back with Range Overs and G-Wagons what are you doing so now you're coming back you're dealing with friends that you know way back then expecting you to do way much and now family cousins all of them yo what did you bring back and none of them ever checked up on you since you were outside when you were away exactly so I had to put all of that in a song that's why I was like I told my mum was like no you stay you just came back to Nigeria after how many years you stay I was like nope booked my flight came to Kenya I prefer to chill with my homies I know from Kenya than to chill with people I barely know half of my life in Nigeria I know right I feel like that is so weird like you expect someone like someone is studying even if they are walking they should come from them from their heart they are like oh I got you this you shouldn't be picky like oh you should buy me a G-Wagons really but they don't know how much as you are wondering no back home we don't have any schools 17 year old is driving a GLK what excuse do you have 17 year old eh yeah oh wow must be like so the pressure back home is a lot in my other life I want to be in Nigeria anyway be careful what you wish okay okay okay I took it back alright I took it back it is amazing by the way what you have been able to do in such a short period how long was the AP in the making it took me it didn't take me couple of months but like yeah it took me they just say yeah just say yeah because people have years it takes years yeah that people take time especially if you are moving from one country to another country that you are not so familiar with I think you did pretty well alright so when are the you know the video is dropping while I was in Mombasa that's why I keep on telling people I didn't go for vacation but most of my friends don't believe that I had the opportunity to shoot with Chef Ali for Nala noodles and kuk studio or whatever and then I was opportunity to shoot two Mizi videos in a day I'm a Mifule and man like Loli what does I'm a mafulemi it is my son name it means it means my kindred will not get lost or my lineage will not go lost so I decided to use it for that song dope as someone said we say hi I'm telling I look for that I think for that comment yeah I'm trying to get their name but you have that though something is it bad boy Jeff yeah I think so bad boy he said like we say hi to you so Salamuzume Fika Justin Kers we say hi alright so where can people connect with you on socials is Loli B L-O-I-B-E on Insta YouTube any platform just Loli B right and your music Spotify everywhere same name Loli B everywhere of course thank you so much for making time to come through I'll give you this time to if you have any last remarks or if you have any shout outs use this camera and do that Peter Thien you know what I always repo moja man I'm telling you umoja whatever umoja shout out to umoja shout out to chef Ali Najib the whole squad you know any device shout out to the whole team it's a privilege to have you guys in my life making this happen so and shout out to DJ F-P-E shout out shout out shout out you guys have the same tip though kinder okay kinder him is gone ham I mean he got to trust Kona he got to trust Kona he got to trust Kona I mean he got to trust Kona he got to trust Kona he got to trust Kona please don't do this please please okay I just like I like my tip where they are but I feel like if I hang with you guys long enough soon anyway you have to shout I'm telling you thank you his name is Lolli B make sure you go check him out he has an amazing EP Miskia what he can do super super amazing and of course mad love to you thank you so much for stopping by


Welcome to the second chunk of the Next Low Advanced Training September 2023 edition. In this first chapter, we're going to be talking about groovy imports and pulling in sort of helpful classes from the groovy and JVM ecosystem to help you accomplish some sort of small tasks and help a bit some pieces inside of your next low workflow. Fantastic. Let's get started. Welcome. All right, let's get started talking groovy imports and JSON passing. As with all the other chapters, we're going to CD into chapter directory. And this one is called advanced slash groovy. Let's have a look around the tree. We see that there's a main.nf and a modules directory. And inside the modules directory, we under modules local fast P, we have a main.nf, which is a fast P module. Let's look at the top level main.nf. We're including that module here on line one. And then we have a very simple little workflow where we define a parameter input by default. Then we use channels from path and use parameter input. We split it into CSV and view the output. Note here that this is not a file on my local file system. This is being pulled in via remote. Of course, Next Low speaks a lot of different protocols. HTTPS, S3 Azure, GCP, blob storage, FTP, all of those sort of fun protocols. What is this main.nf inside of modules?  It defines a container to run in, expects to be given an input of a meta object and reads. There might be a list or a single path. And it defines two output processes, rather two output channels, a reads channel and a JSON channel. Let's just run the workflow as it exists now. Just to make sure this CSV can be pulled in and passed correctly. Great. We have maps. So we have our rows, which are past as meta maps. Well, I'd rather just group the maps. So let's write a small closure here to pass each of these rows into our familiar form of a meta map and then maybe a list of paths. So we're going to take a row and we're going to say meta equals row dot sub map. This is what we learned earlier. Let's just double check that works. Do we have the sample and stratinus keys? The stratinus is not correct. Let's just double check. It's probably just misspelled stratinus. Okay, great. Do we have these meta objects? Fantastic. Two keys. The trick here is that we've precluded the possibility of adding additional columns. So here we're specifying exactly the columns that we want. But let's say we wanted to make our workflow a little bit more flexible to accommodate the ability to add extra columns in the CSV and pass them through the workflow without modification. Here we're being very prescriptive and we're only going to pass through these two keys. So what we would like to do is read this row object, find all of the keys, subtract the keys that we don't want, and retain everything else. So we want to subtract the keys we don't want and those keys that we don't want to retain are those ones that begin with fastq. So fastq1 and fastq2. So let's grab all of the keys. We can do this with the row.keyset method. I'm going to show you what that looks like. The keyset for each row is this. For example, fastq1 and fastq2 is stranded us. But if we add the split method, that will split this keyset into two sets, some that pass this test. So we're going to find all the keysets that match this regular expression. So that begin with fastq, these two keys, and then a different set of those keys that do not pass this Boolean test. That is, they don't start with fastq. That is, sample1 and stranded us. I'll show you what that looks like. If we just remove the destructuring, see exactly here. So for each element, we have a list containing the keys that match our test and the keys that don't match the test. So the meta keys, we'll be calling the meta keys and then a list that we can call the read keys. So I've supplied here links to the keyset method if you're interested in the JVM documentation. And we're also using this split method, which I described, which divides a collection based on some closure, returning a Boolean. From here, we're going to generate the reads. So here, we're going to use this read keys, which is a list of keys. And we're going to put that into the sub map to specify which keys we want. And we're going to collect over the values and return a file object. So now our reads contains a list of those file objects as path objects. So that's ready to pass. Now we just need, oh, actually let's test it because I think this will actually return an error. Unfortunately, it does return an error. It says argument of file cannot be empty. So if we have a closer look at this sample sheet, we'll notice that not all rows contain two fast queue files. Some of them just contain one fast queue file. So for example, this row here contains only one fast queue file. And the fast queue two column is left empty, which means that we are being passed and passing a null value into this file method, which doesn't work. So we need a way of accommodating that. So now, so here we're going to take the sub map of the read keys and the values. What we're going to do is we're going to add one extra method in this chain. We're going to use find all, which is like a filter. We're going to find all those elements where it does not equal the empty string and then return those. So we're basically excluding elements where that element in the row is the empty string. I'm collecting those again with the file off file method, which now works fantastic. Now our reads in these lists, if there are two reads, we're passing two. And if there's only one read, we're just passing a list of one one rate, which is perfect. Now what we need to do is construct the metamap. So we've got our reads object. So we have this meta keys value, which is a list of keys that don't match fast queue. So these are all the keys we want to pass into the metamap. If we look at, if we remind ourselves of what we're trying to shoot for here, we're going to construct a channel that has elements that match this pattern. So a meta object and then path reads. I can see here, if we look inside here, this module requires or expects two extra keys, this ID key and a single end key. We want to be able to pass those through. We want to make a channel that can be passed to break into this fast key process, fast P process. So let's get cracking. Simple start would be taking the sub route of the meta keys. So if we see what that looks like. Again, remember this closure is returning the last, the last sort of expression, which in this case is the meta object. Okay, great. So we have our meta objects contain the key sample and strandedness. And if we were to add an extra column in there, they would appear here inside of our metamaps. We want to add those two extra features. I want to add two extra keys. We want to add a single end key and an ID key.  This question mark equal operators. What this is, is the same as saying meta dot ID equals meta dot ID question mark and ID. So this is saying, if metadata ID exists, then use meta ID. Otherwise use meta dot sample. This is what we call the ternary operator. So if meta dot ID exists, so for example, if there is a column inside of our sample sheet with header ID, then we're going to use that as the meta ID, which is basically going to leave it unchanged. Or rather, if it's null. Otherwise, we're going to use the meta dot sample. We can use the same value for sample as ID. That leaves us one more key to do. We want to have this single end key. So we're going to have a look at this reads object. Remember here earlier that if the reads object contains two keys, that means we have paired and reads. Or if it contains two elements, we have paired and reads. And if we have one object, we have single and reads. So I think that should just not do us. And we're just piping into view before we pipe into fast speed. Just to double check that everything looks okay. Right. Strandedness, auto, single and false and false. Single and true. Perfect. So for those samples where we only have one fast queue file, single end has been set to true. So now we have this channel that matches the cardinality of the input channel to our fast P process. We can now pipe this into fast P. We're going to make one more modification. And we're going to say docker.enabled equals true. So this way we can use this container definition in the fast P process. Of course, fast P process has two outputs, so we can't view all of them. That's going to take a minute or so to run. Note here that I'm always using the test resume flag to pick up case results wherever possible. This is going to be important, particularly now. I can run resume and it should be able to pick up the case results and I don't need to do any extra calculations. Perfect. Seven case tasks. All right. Let's have a look at these outputs. So we have our metamap. Fantastic. And then we have these fast P JSON objects. Let's have a look at one of them. So there's probably some really, there's definitely some really interesting information here that we might want to pull out and supply to downstream processes. Maybe we want to pull pieces of that out and include them in our metadata objects. But we don't necessarily want to write a JSON parser ourselves. We suspect there's probably JSON parser available inside of Guru or inside of the JDM. So let's try and use existing code rather than writing our own. Before we get actually into the JSON parsing, let's cover publishing files. To make our lives a little bit convenient, let's actually add a publish directive to the fast P process. We made a small modification to the next flow config earlier to add Docker. Let's add one more. Inside the process directive. Inside the process block, we can use the with name selector to add a process directive to just those processes that match this name fast P, which is of course the name of our pro fast P process. We're going to add this publish directive. We're going to add a path key. So we're going to say publish all of the outputs from this process to a directory name results slash fast P slash JSON. And we're going to also use the save as directory. That's ready to save as argument to publish here. So this save as argument requires you to supply a closure. This closure takes is iterated over every element that's published or other every output from the process. It receives the file name of the output, and then it returns the file, the rather the name that you'd like to publish that file under. So if you'd like to rename a file, that's how you can do with the save as block. So here we're taking file name and we're asking does it end with Jason. And if it does, we're going to reuse the file name. And if it doesn't, we're going to say null. So basically we're using the save as block as a filter. So we're only publishing the Jason files to this results slash fast P slash JSON directory. Let's try that. Just kind of pause there for people to catch up that test that you can see now here now in our results directory slash fast P slash Jason. We have sim links into each of our Jason files. This is going to enable us to quickly test the Jason passing without having to write waiting on fast p caching. So let's consider the possibility we like to. Yeah, we know we know that there are some interesting things inside of these Jason files that we'd like to capture and use use downstream. There's a link here to the groovy documentation. Specifically the groovy documentation around Jason passing and it notes that we need to import this Jason slipper class. So I can do that. Inside of my main.nf just have this import groovy.json.json slipper. And I'm going to introduce the idea of a second endpoint now. This end points allows to have different workflows within the same main.nf. This can be very handy for. Testing. Fast p. Jason style Jason. And I'm going to view those just to make sure I have capturing them at all. And to run this second workflow note here that it's got a name. I will run next to run resume entry. Jason tests. So after the dash entry argument, I'm supplying the name of the workflow I like to run. Perfect. So I'm returning a channel with each of our Jason objects in here. Let's create a small function at the top of our main.nf that is going to take you responsible for passing our Jason. So it's a little function. We can call it get fields result. It's going to take a Jason file. And here we're constructing a new Jason slipper objects. We're going to call the past text. We're going to call Jason file dot text. So this dot text method is a really handy little groovy feature for reading a file and returning a string of all the text. That will return the fast p result. So I'm going to pause here for a small exercise. Given this fast p result turned from fast p method is a large map. I want you to have a go at modifying this get filtering result to return just the after filtering section of the report. So if we look at one of these files, you can see here there's a small section called after filtering. So how would you modify that function to return justice after filtering section. I'm going to give everyone about five minutes to attempt that. There's a solution here, but try and give it a go yourself before checking the solution. I'll see you in five minutes. Okay, great. Welcome back. I hope you had a go at that small exercise. And here's there's obviously a number of different ways in which you might attempt this. But here's one potential solution. So let's double check. And I'm going to write a little map method so we can check our results. So at the moment I'm returning this whole fast p result. So it's going to return all fast p object passed into a big map. It's going to be quite the both, but let's just see what happens. So I'm not doing any filtering here. So I'm getting the whole Jason objects returned here as a groovy map. So that's a little bit of a voice and a little bit too much information. So let's have a close look at this map. So there's a summary key. And then inside of summary, we have after filtering. So if we want to adjust the summary, we could do fast p result summary. And that returns a smaller object. Okay, so that's not quite as opposed. So we're getting there. The next thing we want is inside of summary. We want the after filtering block. And we can just use dot notation sort of groovy to step inside each of these elements. And here we have just the after filtering results. And that's what we want. So we have some information like the total number of reads, the total number of bases, total number of Q20 bases, all that sort of stuff. So let's say we want to use that to pass into our metamap object. This is pretty close. This is good, but it's slightly dangerous. We're assuming here that the summary object will always exist. And the after filtering object will always exist. We're going to run into problems if the fast p code changes, and they rearrange that JSON objects. Because if the summary object moves or is renamed, we're going to be calling a dot after filtering method on null, which is going to cause an error. Fortunately for us, there's a safe access option. Instead of dot, we can use question mark dot, which means that if this summary is null, then the question, this will not be called and it will just return early a null object, the null object. But if it does exist, then we'll call after filtering on that non-null object. The last thing we can do is we don't need to assign this to a name. We can just call new JSON server dot past it, summary dot after filtering. Fantastic. Great. So we're passing each of these JSON files through our get filtering results method, and we're returning this nice map with all of the statistics from fast p. Now we can use this to join back to our original map. Instead of taking the JSON from fast p, we're going to map it into meta and JSON. Now we're going back to editing the main workflow, the unnamed workflow, so I can drop my entry. So now we've taken the JSON channel, the output channel from fast p, and turned it into a channel that has the meta map, then the map of extra metadata that we've pulled out from fast p. So now we have two channels. We have this channel returned from here, and then we also have the fast p dot out dot read channel. At some point we're going to want to join those back together, and we can use the join method here, which is a new method we've not talked about, a relatively new operator we've not talked about yet, which is sort of like group tuple, but for joining two different channels on a common key. Whereas the group tuple is for joining elements within a single channel, sharing a key. The join method allows us to join elements from two channels that share a key. So I'm going to type that into join, and because I know that the fast p dot out dot reads also has the metadata object. If we look at the fast p objects, I can see here that the reads passes the metadata objects through unchanged. So because we're also passing metadata objects through unchanged, we'll be able to join on that key. Fantastic. Great. So here we have our elements from the channel. We have a metadata object. Then we have another map describing the fast p metadata, and then the fast q dot gz. So the reads from fast p. So these are the filtered sort of trim reads from fast p joined back with the metadata from the fast p passed into a more convenient form. So there's one last exercise here before we end this chapter. Can you amend this pipeline to create two channels that filter the reads to exclude any samples where the q 30 rate rather is less than 98 point 93.5. So we want to have two channels. So we have one channel of reads that pass the filter, and another channel of reads that don't pass the filter that fail the part that the fail the filtering test. We're going to use probably the branch operator that we introduced earlier in the workshop. But I'll give you five or so minutes to attempt this exercise. We'll see you soon. Welcome back. Welcome back. Here's one potential solution. There are a couple of different ways of doing this, but I think this is a neat one. So we have our, we've joined our fast p reads with our metadata. And we were, we have this new channel that has metadata fast p metadata and reads. What we can do is join. It's sort of unnecessary to have these two metadata objects, metadata from the sample sheet and the metadata from fast p. We can join those together. So let's map this. So we have meta fast p, meta, and then the reads. And I'm just going to join those together with a plus operator. Great. So now we only have one metadata objects containing all of the pieces of metadata from those two maps. Great. So one big metadata objects. And then the reads. Now we need to do the branching operation. We can use the branch operator to produce two output channels, a past channel for those samples whose metadata with a q 30 rate passes some threshold and a failed channel for those samples who reads did not pass threshold. We need the branch operator that we talked about earlier in the workshop. I'm going to pass names to this closure. Stabby operator. And again room from the branch operator, we'd give it a name of an output channel and then some expression to return the Boolean value. Because I already have the metadata from fast p in the metadata object, we can just access it here, pull out the key q 30 base. We can say does it equal or is it greater than 0.935 and fail true. So I'm going to use this true fall through. So that if the element or if the sample does not pass this test, then it will automatically go into the fail output channel. And we're going to call that reads. Let's see if there are any reads that fail. There are no reads that fail QC, which is great news. And he reads the past QC. Fantastic. So we have all of our reads that passed QC. If we wanted to pull out all of the values that QC 30 read values of the reads that pass, we can do it with ease dot pass. You know, this is a metadata and then it reads QC 30 bases. That's QC with reads QC 30 bases equals. Whoops. I didn't mean Q 30 bases. I meant Q 30 rate. Excellent. And now that I've corrected my typo, I can see here that we have two reads, two samples that failed the QC that had a QC 30 rate of less than 0.935. So now we could take these two output channels, reads dot fail and reads dot pass and pass them through different routes through our graph. This is one of the great strengths of next flow in that you can dynamically calculate the trajectory of any particular data set through the graph depending on the results from the previous tasks. So you can sort of divert data through the graph depending on results like this fast P quality control test. You might dump the fail results into a data directory and perhaps even warn the user indicating that there were some reads that did not pass QC results and then have this reads dot pass channel that diverts the rest of the data, the good data through the rest of the pipeline. I hope that's helpful. We'll move on to the next chapter. Inside our next flow workflow repository, there are a couple of special directories which are treated differently to other directories. They are the bin directory, the live directory, and the templates directory. Particularly, the bin directory is extremely useful for a lot of next-flow workflows and heavily used in the NF core community. I'm just going to go over some of the details about each and show you how they might be used. Let's get started. Okay, so inside of the next-flow work directory, there are a couple of directories that are treated, especially treated differently to the other directories, bin directory, live directory, and templates directory. Let's start with the bin directory because it's the most important and it's the most useful for most next-flow workflows. Let's CD into advanced slash structure and have a look around. So I have a little main dot NF, which isn't important at the moment. But let's say that I have some small accessory scripts that I'd like to distribute with my workflow.  Perhaps it's being used inside of some of your processes to do some data cleanup or some data munging. It's not an uncommon task to need these small accessory scripts. And there are a couple of different ways in which you might supply those scripts and make them available to you instead of next-flow tasks. The first is to bundle them with your Docker containers. So they could be versioned inside the Docker containers. But for small scripts that might need to be updated sort of semi-regularly, it's a little bit of a pain to have to check them into version control, rebuild your Docker container, upload your Docker container to your repository, and then test your workflow. Fortunately, next-flow has a way to bundle those accessory scripts with the workflow inside of version control. So the workflow and the scripts and the processes and tasks using that rather, so the script and the processes and tasks using that script are all moving lockstep under the same revision control system. So let's give, let's do an example. Let's say I have a small R script and it does some work. The work that the R script is doing is not particularly important.    So it writes two files, a PNG and a TSV. Specifics are not important. This just loads the MPG data set that's bundled inside of tidyverse. And let's say we would like to make a script which uses, rather, a process which uses this script. I'm going to make a little process called plot cars. I'm going to use a particular container that has tidyverse installed. And this script doesn't have any input, but it does have output. I'm going to say it's going to produce one or more PNGs. I'm going to emit them in an output channel called lots. And the same for TSV. And I'm going to say it's going to be emit in an output channel called table. And this script just echoes cars, runs cars. Let's say it could be slightly more complicated. Perhaps we can say echo. Note here that I'm not worried about where that car.r script is located. I'm just calling it as if it was in the path. And this will work both for my local file system or my laptop. But also, if I am running at scale in the cloud, I don't have to care about where that r script is located. In exactly the same way that inside of my processes, I don't care about where the input data comes from. I don't even worry about the path of the input data. Next load takes care of all of that for me. And in the same way it takes care of adding the bin directory to the path. So I can just call cars.r. So one very important thing that I've not done yet is made this car.r executable. At the moment, you can see here, if I run ls-lhbin, I can see my car.r, but it's not executable. So next load will take care of moving that bin directory and supplying it to the virtual machines or the tasks and then adding that directory to the path. But I still need to take responsibility and make those scripts executable. So I'm going to do that with chmod plus x, bin-cars.r, and now they're executable.  Let's run this script and see what happens. The last thing I want to do is modify next load config to use Docker. So in the contractory, I'm going to create a next load of config file. Oops. I could put it in a profile, but I could, yeah, actually let's put it in a profile. So here I'm creating a new profile called Docker and setting the docker.nable true variable inside of that Docker profile. So now I can run next load run, contractory, profile, docker. So what next load is going to do now is going to ensure that the rocker slash tidyverse latest container is available locally, and it's going to run that process. It'll just take a minute or so for that to download while that's happening. I want to make an important note about the shebang line here. I've used here hashbang, user bin, and R script. This is really important for portability. So on my laptop, I might have R script already installed in a particular location. In fact, I do. It's in like homebrew slash bin slash R script. But of course, if I wanted to make my workflow as portable as possible, I'd like to make it available and ensure that I can run it on the cloud. And when I'm running in the cloud, I might not be running inside, like I might be running this R script locally or I might be running inside Docker. And if I'm running inside of a Docker container, the R script location, the location of this binary executable might be in a different location. So if I were to hard code the path here, user bin, maybe. So one option would be to hard code the path here. For example, if I know in this particular container that R script is located in this location, that would work for that particular container. But if someone else wanted to use my workflow and maybe update the container, we use a slightly different version of tidyverse. And if that container had the R script executable at a different location, this shebang line is going to fail. That's why it's particularly important to use a portable shebang line, ideally using user bin end. So that will go the same if you're using R script or if you're using a Python executable. Whatever your interpreter is, make it as portable as possible because this script may be executed in different contexts in a container or different containers or locally or inside a current environment. I can see here that my next task is finished. My process is done. So let's just double check that it actually ran by viewing the output and viewing the PNG output channel. Fantastic. And same for the table. I can see this. So what I've demonstrated here is the ability to use a small accessory script that I would check into version control that would move in lockstep with my next-flow workflow versions. And I can just call as if it was already in the path in all potential contexts and next we'll take care of making sure that's true. So let's have a little look behind the scenes and see what next-flow is actually doing here. So it was this process 46 slash 5C. So I'm going to CD is that directory. I'm going to use code. So this is just that single task working directory. You'll remember that next-flow executes all tasks in their own individual work directory that's keyed by the hashes of their inputs. And I can see here the .command.run file. So this is a file that next-flow writes that takes care of setting up all the data, modifying the path, making sure all of the input data is available. And it's worth sort of occasionally looking through this .command.run to see what's happening behind the scenes. Inside this .command.run we have a couple of important flash functions. One of which is this nxf container nf. And so here we're exporting the path to be the existing path to the nxf container nf. So here I'm adding that bind directory to the path. This works on the local machine. If I was operating on the cloud, this nxf container nf. It would be copying in, first copying in the directory to make it available on the virtual machine and then adding the path. Inside this .command.run you also have staging in, staging out, rather staging in of input data with ln-s. If it's on a local file system or copying it in via cloud command lines if you're operating on the cloud. Again, here is a warning just to be sure about using a portable Shebang line in your open scripts. The second important directory that's treated in a particular way by nextflow is the template directory inside of the root of your nextflow workflow repository. You might have situations where the script block is becoming quite long. If your script block here you might have a long bash script or perhaps you have started a small Python script and you're using perhaps you have some small, rather increasingly large Python scripts in here. It starts small but it sort of grows over time and you feel that this grows beyond a few dozen lines and starts to dominate the main.nf of your workflow. You can use the template directory to spin out that script into a separate file and move it to a template file. I'll show you what that looks like. Instead of our plot cards function we're going to make a new process called say hi. I'm going to use the debug directive that we talked about last time for echoing out the standard output from each of the tasks to the nextflow standard output. I'm going to make a process that expects a channel that's going to supply a new name and I'm going to use a template at a .py. Instead of writing a string here I'm going to call this template method or function and supply the path to a template. You can see here in templates I have this at a .py so this is a very dumb little Python script but you know it's not quite Python because I'm using a full assign string interpolation here so this is nextflow native string interpolation so here this is not just pure Python this is as if I had written this in a script block so instead of writing this in a script block like this let's say I had many lines here it was becoming overwhelming let's just see what it looks like I'm running it in a script block actually just to show you that it works I have each of these names small pipe on my part require a closing braces great so now we have hello hello each of these odd dog names but let's say instead of having this in a script block I'd like to use the templates function so instead of this script block I have this at a .py which is under templates and now I can just simply write template at a .py perfect and so you can see here that we have the same results we have hello for each of these names because this at a .py template it's been pulled out and added in place of my script block the last special directory that we'd like to talk about is the lib directory and this is slightly more complicated you might remember in the last chapter we talked about adding a small helper function so that may not we had that json passing function and certainly for small functions like that json pass was certainly fine and totally reasonable to add them to the main .nf but sometimes those sort of functions grow in complexity and it would be helpful to sort of bundle them into our own groovy class and you can add groovy classes into the lib directory and make them use available for use inside of your workflow both inside of main .nf and any imported modules the number of the reasons why you might want to use this are many and varied for example the nf core irony seek workflow we have at least five different groovy classes that are defined inside of the lib directory for doing all sorts of utility and sort of miscellaneous accessory tasks I've provided the link here to the nf core irony seek workflow and it's worth having just a quick look through the example classes the nf core template uses most of the ones in the nf core are simply provided to execute early in the workflow and provide utility functions like helping pass parameters help texts fancy formatting and all of that sort of stuff but the lib directory can also be used to provide functionality inside of the workflow itself so let's give an example of that let's make a metadata class and so this is just an example problem in almost all cases I'd probably recommend you use a simple map for passing metadata in almost all cases that's sufficient but I just want to use this as an example to show how you might add a class to your next workflow so inside of lib let's make a new file called metadata data.groovy and inside of metadata.groovy we're going to make a new class classes in groovy begin with this class keyword the name of the class and you can because it's jvm and object-oriented you can extend existing classes so I'm going to make a class that extends the hashmap and I'm going to add a simple new method called high which just returns a new string called hello workshop participants so you can think of it just like a normal map just like the many maps that we've been using but it has one extra method called high so we can use this in our workflow I'm going to clean this up we can get rid of our script example I'm in Montreal so I'm going to make a new channel containing city name I'm not going to use it at the moment I'm going to use the new keyword to construct a new instance of our metadata class and I'm going to say hi so I'm going to call a high method on that metadata class let's see what that looks like when I run the workflow remember that high method returns a string saying hello workshop participants great so that's what we see here so we've made a new class called metadata and a file called metadata.groovy inside the lib directory and because our next flow ensures that any classes defined inside the lib directory rather it ensures that this lib directory is added to the class path it means that I can just simply call new metadata anywhere inside of my workflow and get a new instance of that object at the moment we're just sort of making a channel that contains this city name but it's not actually doing anything with it so let's modify metadata.groovy class our metadata class to take a name we can have a constructor so now we're going to have a constructor that takes a location or takes a string but we're going to call location and sets this instance variable location so now this location is going to be available inside of any of our methods so now instead of returning a simple hello workshop participant string let's customize this message by location so if this has a location then we're going to say hello from this location otherwise we're going to say hello workshop participants the last thing we're going to do is inside of this metadata we're going to pass that city name as it if we want we can give it a name that's not necessary I forgot to say the file apologies perfect so hello from Montreal always remember to save your files before executing your workflow so we can also use this method inside of a process so we could pass this through next-door channels into a process so we can have our custom class I'm just going to call it meta so what we're going to do is we're going to echo metadata high so we're going to have next-door resolve this call the high function on our meta object which will return this hello from Montreal string perhaps hello from Montreal and hello from Boston and it's going to pipe that into t just going to echo it to standard output and also write it to this out.txt we're going to add debug true so that we can see it happen on the standard output and here we run next I run oops we forgot to actually put it through here create these new metadata objects and pass them into the use meta process and you can see here we've written hello from Boston hello from Montreal if we want we can do the output from that process which will just be that out.txt we've defined as output here and just to double check you can see out.txt hello from Montreal and hello from Boston so why would you ever want to do this most of the time you won't need to but there will be occasions where it can become very helpful so for example let's say you have some metadata class some metadata inside of a map object and you'd like to add some methods to that metadata object to for example pull out the prefix of an adapter so let's say you have a metadata class that has an adapter key and you want the start of the adapter you could add a method like this getAdapterStart which just pulls out a substring the first three characters of the adapter this is just a very simple example but you can imagine adding extra methods to that metadata class to pull out specialized pieces of information or calculate special values I'll show you what that looks like so let's say in our metadata class in addition to that high method we also have this getAdapterStart and it says if there's an adapter key inside of our map then pull out the substring first three letters create the new metadata and we're going to also this new adapter key so now inside of the useMeta process we're pulling this metadata adapter and the key which was just as normal and we're also using this getAdapterStart method so now inside of out.txt we should see the prefix is AAC which is correct as we described earlier in the workshop these get methods can be shortened but in the context of a map it's probably safer to use the full method name so at the moment we've just demonstrated I've demonstrated using simple methods to pull existing data but you might want even to reach out to external services perhaps to a LIM system or to an API or to some sort of tooling inside of your infrastructure let's add this getSampleName method to my metadata.groovy object and to do that I'm going to need one extra piece of groovy I'm going to import this jasonslip object just as we did earlier for the jason passing example because in this getSampleNames method I'm reaching out to an external URL here in this case it's just a postman code but you can imagine this might be any piece of infrastructure an API or a sequencing machine or any external piece that's communicated via HTTP calls and I'm reaching out to this URL I'm opening a connection I'm getting the response checking that the response is 200 that the response is okay and we pass the jason the response into a jason object or we pass the jason object into a map and then pull out the args.SampleName so we're pulling out a piece of that response this is a very simple little dumb example but you can imagine doing arbitrarily complicated things inside these getSampleNames obviously it's important to keep this not go too overboard you don't want to be doing real computation inside of these things inside of these classes and inside of these methods but it's sometimes very handy to have that ability so now if I modify this process useMeta to add this meta.getSampleName inside so when NexLo constructs the process constructs the task and writes the .commandosh it's actually going to call this getSampleName on the meta object and I can see here it's reached out to the web, called the URL and pulled out the response now there is a warning here caveat when we start to pull custom classes through NexLo we need to understand a little bit more about the caching mechanism when each task is run NexLo calculates a unique hash based on the tasks inputs for example if there is a file input or a path by default it will take the path the string of the path the mass modified date pieces of metadata like that in the case of a value like a map it will take the hash of that map and it will take each of those pieces to calculate a hash for the task as a whole but because we're adding methods to this class that adding methods here is not going to change the hash of the value of the item being passed in which can be dangerous because we might change the way getSampleName works but that's not going to change the hash for the task and so we're going to have caching pull up cache results when perhaps we don't mean to so for the next five minutes or so I'm going to ask you if you can run through this exercise can you show how changing a method in the metadata class does not change the hash of the task we'll be back in five minutes or so okay great welcome back here's one example of the solution so let's say we run this with resumed I picked up cache results expected but let's say I changed the behavior of one of these methods for example getAdapterStart so instead of grabbing the first three bases I want to grab the first five bases of the adapter if I run nextload run resumed a naive approach might imagine that this is going to change the task cache because I would sort of anticipating this is going to change the script because this getAdapterStart method is called inside the script but keep in mind the inputs to the task are just the value itself so it's just in this case because I'm extending hash map it's just the keys and values inside of the hash map which is not changing only the successory method so even though I've changed that method nextload should still pick up the cache tasks cache will only change and be recalculated if I change the values inside of that and hash map so if I add if I change the hash values now nextload will recalculate the hash hash will miss and then it will use the new getAdapterStart method but changing the methods in and of themselves do not change the hash so in this example I've shown you extending an existing method existing class this hash map class but we could also go on creating new classes for example let's create a dog dog groovy a dog class and this class is sort of like a records object it contains a string name and a boolean value it's hungry let's simplify our workflow so let's create a new dog where his name is phyto and we're just going to log in folk found a new dog and then showing the dog oops we run this fantastic found a new dog and then it's returned the string representation of this dog object we can pass objects through channels so if we change our workflow to have a channel of dog names and then we pass it through a map where the closure creates a new dog objects with the name and we view those outputs we see our three new dogs but at the moment we're missing something before this can be used inside of a next flow task and caching to be used correctly so for the next five minutes I'm going to pause if you could run this exercise and show that the dog class is not cached when resuming a workflow we'll see you in five minutes okay welcome back make a process that uses one of these dogs to show that the caching is not working dogs have a name property which I can address like this I can pass each of them to the pat dog process and have run it three times now if I resume not change anything the note here that no caching has been used for these processes to do so we need to turn this dog into a value object so next loop very hopefully has provided a decorator to serialize these custom classes by adding this value objects decoration it will use the sort of properties here the name and the is hungry value as the elements that are being hashed into the to provide the inputs to the task the last thing we need to do is register the class with car cryo which is the serialization library I'm going to do that inside of the main.nf ideally at the top so I can just do it here importing the cryo helper and then cryo.register dog so now last exercise for this particular chapter is I want you to show the dog class now that we've done the work of adding this value object annotation and then registering our class with cryo helper that can be used inside of processes and the caching works correctly see you in 5 minutes welcome back let's try now with this new cryo register dog class so now if we use this pass this dog to this pat dog process first time registers the caches and now if we rerun the same tasks we have 3 cached tasks so now that we've registered this dog next load knows how to cached this or sort of calculate hash for this dog object so in this case because we're using the value object it's going to take the name and the boolean values so these classes here the class values here and use those as hash inputs so that if they change we should see two cached tasks and one new task perfect great so this is how you might introduce custom classes into your workflows for small utilities or reaching out to external services from here we're going to take a short break if you're watching this live now is a great chance if you haven't already to drop by the septum set 23 advanced training channel on the core slack myself and other volunteers are hanging out there if you have any questions we'd love to help out if you want to chat anything next low or NF core we'd love to talk to you there of course very welcome to pop in questions all the way through the workshop but we'll see you there if not we'll see you back here for more training in half an hour see you soon we want to cover a little bit about next low configuration well might not be the most glamorous of chapters particularly finish on it is a particularly important piece of how next low works and something that we think some people can get wrong quite easily so we're going to talk about where configuration is set order of precedence and cover some sort of intermediate configuration options about dynamic directives and label selectors let's get into it so let's talk config it's an aspect of an expert that can be a little confusing to some there are multiple ways of learning configuration it's very flexible which is a great advantage but can be a little intimidating for some newcomers gives us a couple of complications like particularly at which location should I be learning a configuration value and what sort of configuration can I change or what sort of values and what sort of behaviors in next float can I change by configuration the first things I think we need to cover is precedence but before that let's make a new directory I'm going to call it advanced configuration let's talk about the order of precedence what I like to how I think about it is that the precedence is really roughly in order of distance from the command line invocation so those parameters specified directly on the command line like parameters with a double dash notation or configuration options with a single dash notation take the highest precedent you're typing them on the command line they're right there at the command line the next highest level of precedence is the parameter files or parameters that are supplied by a dash params-file option so these are again supplied by the command line but rather meet directly on the command line referred to by a command line option similarly the configurations the sort of twin, the cognate config option is the dash C option so params-file for parameters and dash C for configuration so parameters are those parameters that are set by the workflow and the dash the config is configuration that affects next float as a whole the fourth level of precedence is the file named next-floater-config in the current working directory so these are not referred to by the command line it's sort of implied there's configuration values when next-floater-run it looks for a next-floater-config file in the current working directory and picks up configuration of parameters from that file that's the fourth level of precedence one level further away is the next-floater-config in the workflow project directory so that might be a next-floater-config in GitHub somewhere or GitLab or bitbucket or code commit or something like that so it's the next-floater-config in the workflow project directory so in the root of the workflow or a configuration file that's imported by that file by include-config almost second to last is the configuration file in home so home.next-floater-config so these configuration values apply to all runs for a given user but they're very low precedence and lastly the final configuration values are specified by the workflow itself in main.nf or any of the .nf scripts inside of the workflow so you can think of it as stepping away from the command line from right on the command line to a file referred to by the command line application to a file in the current working directory to a file on GitHub or a remote server and then finally parameters configuration in main.nf on the workflow in the remote server so why might you use some of these configuration values for example system-wide configuration these are really handy for setting configuration values that you know you want to have applied on every run on a given system that is for example I might want to use Docker on every run on the system like even on this particular machine I have Docker, I don't have tools installed so if I run the .nf workflow it will take a second to download the containers but I don't have any of the tools installed so I will always, this will certainly fail because I don't have for example fastqc installed or fast yeah fastqc in this case so I'm always going to want to run inside of Docker so I'm going to run docker.enable equals true into home.nextload .config just to make sure it's there and now I can run exactly the same command next we'll run rnc.nf and it's going to use the Docker containers it's going to pull and use those Docker containers for each of these processes and I didn't have to change anything on my command line indication because those configuration options specified in home.nextload.config will apply to all runs unless they're overridden by configuration at a higher level of precedence another reason you might want to do this is for example you're on an HPC system that always has Slurm as the executor and perhaps always uses Singularity for their containers your home.nextload.config file might include lines like these process or executor is Slurm and Singularity.enable equals true the configuration values will be inherited by every run in that system so let's talk about overriding process directives process level directives like CPUs memories there's a whole list of them here in the next load configuration there's many many configuration process level directives all of these can be overridden by by configuration so let's give an example let's create a new nextload.config in the current working directory and we're going to say process CPUs equals 2 so I would like to make sure that every single process every single task that I run in this working directory uses 2CPUs so I can run that same nextload run at RACGNF the same workflow and it's going to ensure that each task is supplied with 2CPUs and that's going to be all of the tasks unless otherwise overridden we can make these configuration values more specific by using process selectors so we can select the name or any labels that have been defined by the workflow authors so I'm going to run again a process block and this is in nextload.config using the with name selector I'm going to make sure that any processes or any tasks that match the process name RACGNF that is this task here are run with 2CPUs all other tasks are left with default I can override the default with CPUs 1 so this says this ensures that by default all tasks get 1CPU unless they match this with name selector in which case they get 2CPUs and if we have a lot of tasks we might also be interested in using glob pattern matching so instead of RAC colon index we could do dot star colon index to match any tasks with a process name that ends with index just to make sure this is working it may be helpful to set the tag which is the tag is the value that's printed out in parentheses here of course I could log into the CD into the work directories and double check the CPU allocations but sometimes it's quite handy just to set the tag and I can see here the tag for the RAC index has been set but the others have been remained as the default so this is interesting but the really powerful work comes in dynamic directors so you can specify dynamic directories using closures same sort of closures that we've been talking about for the whole workshop and those closures are evaluated as the task is submitted there's a mix of next flow that is a data flow oriented sort of paradigm in the task execution is calculated at like there can be there can be elements of the task that are calculated as the task is executed rather than before when the run begins so your next look can be very dynamic and calculate directors as an example of what tasks submission time so let's say we want to run the RAC again the run we've been demoing here that includes a fastqc process that looks like this so it's the process fastqc and it has some inputs the sample ID and path reads I can already see here that we're dynamically evaluating the tag this sample ID is being used in the tag directive here with dollar sign sample ID we can override that tag and if we're using setting values inside of the process like a .nf script we don't need the equal sign but it's really important to note that if you're setting values inside of configuration the equal sign is needed the other thing to note is that if we're using dynamic directives we're going to need to supply a closure here so we're saying I'm going to defer evaluation of this I'm just going to supply a closure that's going to be evaluated later rather than evaluated when the configuration file is passed I'm just going to change the wording here to found a sample fastqc, found a sample ggalgal so in addition to the sample ID value we also have access to the reads value so we might want to dynamically scale for example the number of CPUs on a task depending on how many reads were supplied again because we're going to use a dynamic directive we're going to put it inside of a closure I'm going to say the CPUs is value equal to reads.size so if there is only one read supplied then the task gets one CPU and if there are two reads supplied then the task will get two CPUs just to make sure that I'm actually doing this we can set a tag to pull that reads.size evaluate that submission time as well perfect, an even more advanced option that I haven't seen a lot of but I suspect we might start to see some is scaling the resources like CPUs or like memory based on the input files size not just the size this size here is calculating the size of the collection so how big is the list of files but we can also calculate the size of the actual files themselves in bytes so here let's make this tag directive a little bit more complicated so the total bytes we can do arbitrary computation here the total bytes is the reads and we're going to use the spread notation that we introduced earlier so we're going to iterate over every element in this reads collection and calculate the size which is going to return the number of bytes and we're going to echo that out here or rather we're going to change the tag to say the total input size is and we're going to use as memory unit and a memory unit is a next-low built in class for turning strings and integers into human readable memory units and we need to actually sum those to sum those together so when we take the reads collection we calculate the size of each of them that leaves us with a collection of integer sizes and then we call sum to sum them together you can see here the total input size is 1.3 makes so you could use just as we're setting tag here you could use the same sort of idea to calculate the total input size of a given process and use that to scale for example the memory or the CPU use of that task or process you wouldn't necessarily want to use that for all tasks as a blanket statement but if you know particular programs or particular steps in your workflow scale or the memory required scales with the input size then this might be a helpful way to dynamically calculate the amount of memory that is required by a particular task and lastly the most common sort of dynamic directive is used for retry strategies next-low gives you the option of if upon task failure resubmitting that task again you might want to resubmit it again because it's a flaky program that requires network IEO or you would just like to give it another go but you could also dynamically evaluate things like CPUs and memory so that on resubmission the task is allocated more memory or more CPUs to do this two directives are needed the max retries and the error strategy directive the error strategy directive determines what action next-low should take given a task failure that is if the .execute file does not contain a zero so the options are either you can terminate the run completely and this is the default so upon any task failure next-low stops everything it cancels in progress tasks and winds everything back cancels and sort of finishes and exits as quickly as possible the second option is finish this is a little bit more elegant and the next-low will allow processes and tasks that have begun to finish on their own steam and then tidy everything up but it's not going to submit any new tasks the ignore retry strategy the ignore error strategy simply ignores any errors from that task on that process and it simply doesn't output anything into the output channel this can be a little bit dangerous but there are times when it's helpful and the retry error strategy is the one that will allow next-low to resubmit the task and it will resubmit the task up to a value specified by max entries so if we're using a closure to specify a dynamic directive we also have access in addition to the variables defined like reads and toe like the ones here sample ID and reads we also have access to a special variable called task and this task variable has an dot attempt property and that dot attempt property is incremented every time the task is retried so this allows us to work out for any given run and what is the number of task attempts so here we can imagine setting the process configuration like so so here we're going to say for all processes with name RNA sequence we're going to retry on any failure we're going to retry up to three times and the memory taken by this task given to the task will be defined by the disclosure so on the first attempt next level attempt to run the task with two gigs of RAM and one hour of wall time on resubmission it's going to take four gigs of RAM and two hours of wall time and finally it's going to try six gigs of RAM and three hours of wall time as before I mentioned before when using config configuration via this next way dot config or you will always require equal sign and as opposed to specifying configuration and process directives inside of the NF file where the equal sign is not required I hope this has been helpful and I look forward to seeing some interesting configuration values popping up in the NFCore community soon with that I want to make sure there's plenty of time for Q&A we have a lot of people who have registered for this and I expect there will be lots of questions and just so we get them we're going to pause here to open up the floor on Slack so don't forget to log into the SEP23 Advanced Training Channel on the NFCore Slack ask your questions there and I want to thank you for coming and listening to the first community advanced next-flow training we'd love to hear your feedback and we look forward to seeing you at a future event thanks very much I'll see you soon