Hello, everyone. My name is Franziska Bohnert. I'm today's host. And with us is Chris Hackard. And he is talking about how to implement custom scripts into your next flow pipeline. After you. Awesome. Thank you. And thank you for the introduction. So, yeah, today I'll be talking about custom scripts and how you can add them to your next flow pipeline. I think this talk was inspired by questions that we see on Slack occasionally where people are having trouble implementing a Python script or an R script or a Perl script or some other script as a part of their pipeline. And quite often there's not necessarily mistakes but things that can be done to make pipelines a little bit better and more readable. So today I'm just going to try and outline some of these things and hopefully everyone will be able to walk away with a little more understanding of how to do this with next flow. Next slide. There we go. So today I'll be talking a little bit of background. It's kind of outlining the problems and how we can solve them. I'll introduce my first pipeline, which is a next flow script that I've very quickly written to sort of demonstrate how to use the bin and templates directory to store your custom scripts. I'll quickly talk about managing dependencies and some of the things that you might need to consider when you are sort of packaging those together as well as a really quick summary. So background. I don't think it's a secret that in real world pipelines, you often need to use custom scripts which can be written in different languages. So bash our Python Perl as well as others. With next flow, you can integrate any scripting language into your workflow by adding the corresponding shebang to the code blocks or the script block. And I'll demonstrate this really quickly in the next couple of slides. And you can avoid keeping large code blocks in your main workflow by executing them as custom scripts. So while most scripts, some scripts can be really short, some can be really long and to improve readability. It's recommended that you store these elsewhere and then execute them using next flow rather than just having a big, big release of troublesome code block, which can be quite difficult to get through if you are trying to read through someone else's code. So this is my first pipeline, which is a next flow pipeline that I've written. As you can see, it contains a single process called my script, which is going to take string values as an input and give standard output. In the script block there, you'll see that all it's doing is taking the strings defined with the name str from my input. And it's going to turn all the lower case letters into uppercase letters. So nothing too complicated. So this is a really simple, in this case, single line of code. In reality, your script could be much, much larger and also written in a different language. In the workflow block down the bottom here, we have this, that and other being the three string inputs that we're taking from the channel. I'm piping that into my process and then just using the view operator to show that in my output, in my screen terminal window. So as I mentioned earlier as well, you can just easily add shebang to the top of the script block to change the language of that script block. In this example here, I've changed the script from sort of bash to an R script. And what I've done here is just re-written my sort of my job or what I was trying to do, which is turning lower case letters to other case letters using this to upper function, which is a base function in R. And then just catting that. So I'm printing it out to the screen. Nothing else has changed in this pipeline, apart from this here in the script block. I just wanted to point out that shebang again, because in this case it's an R script, but you could also include Python, Parallel, like I said, any other script in this. Any other shebang to decide which type of scripting language you would like to use. So this is just a really short animation of what this pipeline is actually doing. And as you can see, it's just running the pipeline and printing that other in this all in capital letters. So this is just those three strings that I've included there being printed out to the command window. And just once again, so it's just executing that pipeline and printing those out. So while this is a short single line of code, if it was much larger, you might automatically think, okay, I want to remove this and have it as an executable R script somewhere else in the pipeline or someone else on your system. And this is what I've done here. So I've decided to call my code block, my first script R just because it's an R script. I have changed this slightly, which I'm using command arguments to allow for an input to be taken as this script is executed executed in my script block. So that's what's happening up here. We've got my first script taking the command arguments. True. So it's going to take the trailing arguments and absorb them as a part of the script. Down here in my pipeline, what we'll see has changed is that we've got this full path to my first script. And then it's using this str, which is the input or the named input as part of this process. So what I've done here, and this will work on my system, assuming that full path to my first script is a real, real file path. This will work on my system, but it's not overly portable with next flow. I think one of the greatest strengths is the portability of the pipelines. So there are other ways to sort of do this so that if you were to try and share this pipeline with someone else, you wouldn't need to hard code in this file path again. You would just be able to do this automatically using using. Which I'm about to talk about. One thing I did want to point out here is that if you are making a script from your code block like I have done here, you do need to make sure it's executable. So you need to run this on your code to make sure that it's got the right permissions. So the first way that you can store your scripts is using a bin directory. So instead of trying to sort of include your first, my first script as a part of the directory, the same directory as your pipeline, what you can do is create a folder called bin and then store your scripts in there. So what next flow does is whenever you execute pipeline, it will look for the bin folder within the directory of your pipeline. And if it's there, it will mount the files or the scripts in that folder to your path and they'll be automatically executable in your pipeline. So what that will look like is here you've got my first script and the bin directory. This isn't the same directory as you'd expect your pipeline to be in. This is the same script as before. But the only difference here is that it's got a different, you're not having to specify the whole file path. You can just have it here in the bin directory.  And what next they will do is automatically, like I said, mount this file, mount this script in your bin, and it'll be executed executable automatically, which is a really powerful way of storing the script. You can have lots of different scripts in a bin directory and they can all be executable automatically. There's also another way to do this, which is using the templates directory. So this is very similar to a bin directory and that you can have templates, a folder called templates next to your pipeline file with your script stored in there. Now there are a couple of differences that you might notice while of course it is in a different folder such as in this templates directory. And here I don't need to specify arguments or the arguments command that you saw previously because what next they will do is treat this. I guess it'll treat this exactly like you'd expect a script block to be specified and what it will do is using this template. It'll just look in the templates folder and execute this as if it was a code block included here in the speechmarks. What you'll also notice is that here I've included this named input and it will automatically be able to use this straight away. You don't need to use arguments like you did with the bin directory. So the next thing is dependencies. So the scripts I have been executing worked because I've r-installed on my system locally. But if you are using it on a different system or you want it to be 100% reproducible on a different system, you will need to consider dependencies and how they're managed. So dependencies with a custom script are managed in the same way as other tools and I'll show this very shortly with some examples. But of course with a custom script like this you might expect to have one or more different tool packages which can add complexity to how these are sort of integrated and stored. As with other tools, if you are using multiple tools in the same module or same process, you can store them in a combined mull container. While I won't go into it today, there are helper tools and documentation available. These slides will be available and both of these are clickable links where you can sort of read a little bit more about this, about how NFCore has a modules mull function that can help you find mull container with the dependencies you're looking for, as well as how to package multiple containers in one mull container. So this is an example of a mull container, a mull container of how to use dependencies with a custom R script. So this is directly from the RNA sneak pipeline. The process is called stamina summarized experiment. And what you will notice is that down here in the code block we have stamina summarized experiment. This is an executable R script with a couple of argument inputs. And at the top here we have the condor and container declarations. In this situation it is just a single R package and tool. So this is R-basis a part of this package already. So you only need to specify this once. And this is also already have the Galaxy project and biocontainers images containers available. So this is relatively simple example with just the one tool. But you can also have an absolute monster. Also from RNA seed pipeline here we have the DC2 QC process. And what you can see here is that we have a very large number of different tools that are implemented as a part of this. In this case, a mull container has been created, which contains all of these. This is kind of a, probably not a great example because the versions of these tools haven't been pinned to the condor tools. And that's because there were conflicts as this is getting created. But normally you might expect to sort of see some version numbers after each of these. Again, this is probably a bit more of a monster of a script. I haven't shown this here, but you can find it by going and looking into the RNA seed repository. And what you can see here is that this is an executable script again with a number of different inputs that can be taken as arguments. So just kind of summarize what I've covered today is that next flow can use custom scripts written from many different languages. Scripts can be stored in both the bin or the templates directory and both of these will be available to the next flow script. Meaning that you don't need to specify an absolute or a relative path as you're executing a script. And it's really fantastic to do this because it makes your scripts much more portable and usable by others. Dependencies can be managed using condor and containers in the examples I've shown. You can see that it can be quite simple or much more complex with the use of multi containers to help you store all those together. The singularity and condor images. And with that I will finish. I think we're probably about where I thought would be for time and I'll finish on this. And if you have any questions, I'm happy to do my best to answer them. Thanks very much. Thank you very much. So anyone can now unmute themselves if there are any questions you can just ask them or put them in the chat and I will read them out. There is one question in the chat. If you're on a newer next flow version. There's also and a link to module binaries. Yes, exactly. So I've included probably some simple examples here using the bin and templates directory. You can also store these scripts another way in the along with modules. But I haven't gone into that as much through his documentation on this on the next play website. But today I just wanted to focus on probably what I think are the more easy examples, the simple examples, but you probably don't need to do the more complicated systems with complicated techniques. Okay. John is potentially asking a question. Yes. Yeah, we can hear you. Great talking. I was just wondering if there are small scripts or routines that one use often and in many different pipelines. Is it possible to put them on a guitar repo and then pull them in or to have a script stored some common area. Good question. So I think at the moment, most customer scripts are stored locally and executed locally. So compared to integral modules, which kind of the shed where you can sort of download them directly. Most examples of custom scripts I think are stored locally, but with as I mentioned very briefly just before. You can store scripts locally alongside a module with templates and that could also be downloaded at the same time as a module. I don't think that's really being done yet. But there's nothing stopping you from sort of copying and pasting these scripts into your own next slide pipeline and executing them directly from the bin directory or the templates directory. So I guess I guess the bottom line is not that I know of, but it could be done. And there probably are examples out here that people have done this. I just don't know what they are. There's another question on the issue of portability when using custom scripts. Do you mean if the custom script is not within the directory tree or the base dear. So with a custom script, the next flow stages, all of your inputs for execution, if the script isn't defined or isn't included, it won't necessarily be found. So if you were just to include a script alongside your main next flow script, it wouldn't be staged because they wouldn't be able to be found when their pipeline is being executed. So if it's just in the pipeline directory, no, it won't be found. But if it's within the bin folder or the templates folder, then it can be included and will automatically be sort of staged because it's in the path of the script or of the tool, which is a kind of some sort of question again. Yes, if it's not in baster, it won't automatically be found, although you could probably use that to specify a relative path. So I wouldn't necessarily recommend that when you've already got the bin and templates folders available for you. Okay, it seems there are currently no more questions. If you have more questions, you can always go to Slack and Bite Size channel, or you can contact Chris, I guess directly. I'd like to thank Chris and, as usual, also the John Zuckerberg initiative for funding these talks.


Kharkiv at risk of becoming second Aleppo, Kharkiv Mayor Ihor Terikov believes his city is at risk of becoming a second Aleppo if left without help to obtain air defence systems according to an article published by The Guardian. Russia recently intensified attacks against Kharkiv, which had a population of 1.4 million in 2021, with the use of missiles, glide bombs and drones, destroying energy infrastructure and killing civilians according to Kiev Independent. Terikov said the US Congress needs to pass the delayed $60 billion Ukraine aid package to ensure new supplies of air defence and to prevent Kharkiv being a second Aleppo, referring to the city in Syria that was devastated by fighting during the Syrian Civil War. The cost of rebuilding everything destroyed or damaged in the city has already reached More than $10 billion, Terikov said earlier in April. Western officials believe that while Russia lacks the capability to launch a fresh offensive on Kharkiv, Moscow is making a coordinated effort to cut off supplies and create conditions that make the city uninhabitable. Bloomberg reported on April 16. Bloomberg's report echoed Terikov's remarks to the economists that Russia aims to make the city uninhabitable for civilians. By the end of March, Russia destroyed all the electrical substations in Kharkiv, leaving Ukraine's second largest city without a stable power supply. While Kharkiv is at particular risk because of its proximity to Russia, lying less than 30 km from the border, stocks of air defence are low across Ukraine. President Volodymyr Zelensky confirmed on April 16 that Russia managed to destroy the tripelier thermal power plant in Kiev Oblast in a missile strike on April 11 because Ukraine had run out of missiles for its defence. Russia may transfer fighter jets and air defence equipment to Iran. Russia is considering providing Iran with modern fighter jets and air defence equipment to protect it from Israeli and US attacks. Military cooperation between Moscow and Tehran has reached a new level, reports The Washington Post. According to the data, Iran has opened a new dangerous chapter in its relations with Russia by agreeing to supply thousands of combat drones and missiles in 2022 to help Moscow in its war against Ukraine. The expanded ties helped cement deals between Moscow and Tehran, including Russia's promise to provide its ally with modern fighter jets and air defence technology that could help Tehran bolster its defences against any future Israeli or US airstrike. It is unclear how many systems have been delivered and deployed, but the Russian technology could turn Iran into a much more formidable adversary, with enhanced capabilities to shoot down aircraft and missiles, officials and experts say. The arms deals, some of the details of which have not been previously disclosed, are part of a broader cooperation that includes joint production of military drones in Russia, sharing of jamming technology and real-time assessments of weapons deployed against NATO and in Ukraine, intelligence officials and arms experts said. According to them, this cooperation brings significant benefits to both countries, while elevating Iran's status from a junior ally to a strategic partner. Intelligence officials said that Russia is pushing forward with secretly concluded deals to supply Iran with Su-35, one of Russia's most powerful fighter bombers. On the defence side, Iran has long been asking Russia for the latest anti-aircraft missile batteries to protect its nuclear and military facilities from a possible US or Israeli attack. In 2007, Tehran signed a deal to buy the Russian S-300 anti-aircraft system, but Moscow postponed the delivery of the weapon due to pressure from the United States and European states. The voluntary ban ended in 2016 and Iranian S-300s became operational in 2019. Since then, Iran has sought to acquire the more powerful Russian S-400 system, although it is not publicly known whether Moscow has taken steps to supply S-400 batteries. Some S-400 variants are equipped with radars capable of countering stealth technologies used by modern military aircraft. According to intelligence officials, Moscow also benefits from this cooperation. In addition to the thousands of drones purchased from Iran late last year, Russia agreed to buy about $2 billion worth of additional military goods, including anti-drone defence systems which have become a top priority for Russian generals in Ukraine.


Today we're talking about CBD hemp production, and CBD hemp is a relatively new crop for North Dakota. It's a crop that requires a lot of labor, physical manual labor, unlike our traditional cropping systems. So we want to just talk about some of the issues here. CBD hemp we want an accumulation of CBD for the oil content, but we also want to limit the THC content. And so we're trying to figure out how to do that. And we've put together a series of different trials to look at CBD accumulation and also THC accumulation within a plant. We're working with planting dates to see, you know, not only CBD and THC accumulations, but also biomass or yield is part of what we're looking at. And along with that, we're looking at some fertility issues. So what are some of the fertility requirements to enhance CBD production and to limit THC production or accumulation within the plant? And also, again, yield is a big component of what we're looking at. We're looking at plant spacings, plant populations, you know. So how close can these plants be growing close together? Can we plant them in 30-inch rows or do we have to go, do they do better in 5-foot rows or 6-foot rows? You know, and also the plant spacing. So those are all issues that we're looking at. We'll also be looking at harvesting or the timing of harvest so that we can try and track the THC and CBD accumulations. You know, what's the proper time to harvest this? Now obviously, if the THC level becomes too high, then the crop has to be destroyed and along with that investment. Thank you.


Now it is my pleasure to introduce one of my favorite people in the sharing economy and a great thought leader She has this incredible way of making the incredibly complex simple and incredibly apparent Please welcome me in joining. Please join me in welcoming to the stage Author of the mesh why the future of business is sharing Lisa Ganski Hi, good morning. How great is this? Thanks for joining us in San Francisco. Just out of curiosity. How many people are not from San Francisco? Awesome. Oh my god, that is so great. Thank you for coming. This is so great So I'm gonna do a couple of quick things One is thank you very much for I'm looking forward to the next few days Did I do something wrong already? Possibly Uh-oh Hang on Is there a little technical glitch maybe while she's doing that? I was just gonna say I was gonna take the time to do three things One is kind of a super fast in case you you forgot some of the key pieces. Thank you so much But it's not going up there I Could turn this around but that seems Not really technically sophisticated and it is San Francisco. I'm not worried so three things So the first thing was I thought we could do like a really fast in case you're new to the sharing economy a little like You know the top is version Top this I said not topless if people are time The second thing is just to say It turns out that that movements and economies and companies like bio biological things have life cycles So I thought it would be a good thing to answer the question of where are we? with respect to So whereas our movement is it in its early infancy and and how do we know and then the second part is You know, there's been some real trends that are very clear for the first few Big companies for example five billion dollars have been raised by venture capital that gets, you know The lion's share of attention by press Five companies have actually got two billion dollars of the five billion and so in that case We're looking at things. This is part of the sharing economy. We share islands Yeah, so anyway, I think the the bottom line is just to say that for us From my perspective as I look at the arc of what's happening so much of what's been gaining attention Are things that have raised a lot of money or scaling internationally? Which are amazing and great and certainly have validated a lot of what's taking place But on the other hand we see things that are happening at the grassroots level peers is is a you know The master of of corralling that sort of thing But a lot of that is also happening by community funded things Capital campaigns like Natalie said in Kansas. I thought if I mentioned Kansas, there'd be applause because that's what happened with her Kansas, okay, and then and then things like cooperatives, which we'll be hearing a lot about a little later. So Here's the the speedy version last century the 20th century Was what I call the century of the generals General motors general dynamics general mills general electric It was top-down One to many. Hey do what I say we got this handled Followed a military model and our organizations and a lot of our thinking including our education system follows that model This century the 21st century is a totally different theme We have different tools challenges ideas stresses opportunities At least the first half of this century is very much organized around We were on a roll for a second Hold it sharing Okay, we share lots of stuff sharing isn't new to us, you know, we share parks and our transportation Definitely beers at pubs Please don't tell my insurance company I took this photo, but it looked really cool at the time we share infrastructure Weirdly we share tips about how to clean aluminum siding and We share so many things that are happening at the urban level that Basically, you know Education and food and how we live and how we work So much of what's happening in the collaborative economy is happening in cities So cities are platforms for sharing For me and it's been mentioned already One of the main tenets of the collaborative economy is this that we're moving from an orientation where we aspire to own things as individuals and as companies To one in which we want to now either own in different ways But mostly we're moving to a world in which access to goods Services and talent will triumph over the ownership of them One of the great things that is happening is actually technology has brought us except for this little device apparently but Technology has made us more connected to more people on the planet than ever before But in addition to being technologically connected we've learned again Thankfully in recent years how powerful how valuable how Wonderful it is to be connected to other people and the core of building resilient communities and resilient brands as we'll hear it Shortly is about being connected and reaching out into the communities and into the crowd Part of what's driving this is Population we as of this morning or at seven point one seven bless you billion people on the planet today We're moving towards nine point six billion in 2050 this is like a big number 75 to 80 percent of these people are expected to live in cities and we're already Seeing and feeling so much of that pressure on cities most cities aren't in condition to take care of it Climate change we several decades ago had single-digit disasters in the world in 2012 we had 1553 there's a lot of pressure on companies and communities to Kind of create new muscle memory and become resilient in new ways and so much of that Happens from us being connected to each other The internet of everything or the internet of things or the industrial internet It has all those names is the idea that not only are we connected to each other But our things are connected to each other Connected cars and homes There are sensors and streets in San Francisco that know how much traffic is going around and could raise the price of parking on street All of these things are enabling us really from a technical perspective To create new kinds of business models and opportunities so that we get more value As a global community out of what we already have known and then lastly the recession in 2008 Really kind of caused a lot of people to say holy crap. What's going on here? I have to rethink how I spend and what I value Ultimately, I think that that nestles down into this little phrase which is unused value equals waste that Increasingly we've become aware that the things that we value if we're a business on the asset side of the balance sheet Or if we're individuals the stuff in closets and garages Talent what we're not using in this moment is wasted and because we can now take things that were invisible like My bike that's sitting in the garage for three months because my nephew is not here or my kayak Those things are sitting around wasted. Well now we can reveal them and they become part of a new kind of marketplace one of the things that we've seen travel is an industry that's taken a big opportunity here to Rethink the way travel happens and surprisingly perhaps to many of you private homes are One of the things that's growing fastest in travel air B&B, which I'm sure many of you have heard of Is the leading platform? This also is I'll take the second to say a peer-to-peer marketplace Which is essentially on both the buyer and the seller side the traveler and the place where you stay They're both individuals. They're both peers You're not buying from a big company and this model a peer-to-peer marketplace or a two-sided marketplace You'll see that model basically being shown and discussed throughout the next two days So Airbnb to give it some context has 600,000 rooms in six years the largest hotel chain on the planet the intercontinental has 650,000 rooms in 663 years So in 10% of the time almost as many rooms the other thing so peer-to-peer marketplaces are very Easy faster and more flexible. Let's say to grow and to be responsive But also from a resilience perspective when there's a demand surge because there's the World Cup in Rio or there's You know a huge issue called the hurricane Sandy in New York These marketplaces are much more adaptable to the necessities of communities than the other capital-intensive very structured completely owned hotels so one of the things that we see as as Movements and economies mature is we impersonate each other We actually get good at copying things that work and you'll see the two-sided marketplace and the model where? The platform takes a percentage for bringing actors together buyers and sellers That's a model that you'll see repeated and refined as we go Here's an example, which is a company started in the UK called fanbed. They were really big football Fans whenever there was a big game there wasn't there weren't places to stay So this is a community Organized around sport and you'll see this model applied hundreds and hundreds of times all over the world We're learning rapidly from each other as you heard earlier and also some of the research and the general feedback from these Marketplaces is that we love being connected to other people that there has been a joy and a liberation in actually being connected to individuals What I think a big part of what's fueling what's happening and the speed of the movement is the culture of generosity The characters to the right or the we share community who are conspicuously generous and trying to Enable all sorts of things. There's another kind of generosity that I refer to as open-air experiments That's to the left in that case believe which is in 2007 the first big bike sharing service since that time over 500 cities have launched bike sharing services all standing on the shoulders of what we were able to learn from the open-air experiment of the lead Aloja Vasia Aloja Vasia in Portuguese it means empty shop This is another great example where last winter in Sao Paulo where it's quite cold even though Brazil is You know a hot country in in more ways than one They started with an empty shop and the idea in a very conspicuous location was Bring really beautiful things to donate to other people and so the shop filled up over time They rate they had three hundred and three point two tons of Clothing donated but also it was such a conspicuous and such a beautiful model that they have decided to open source it and now Aloja Vasia or empty shop is happening all over the world So this is one of the things also the open-air experiments and the open sourcing have been really one of the ways that we've accelerated the movement and learning from each other and sharing a Gentleman who's over decades has done work on this idea of the Maturation model for corporations and movements is dr. Adiza's it's a long conversation But the short version is I think that we are in phase one of the collaborative economy between go-go and adolescence We know this And it's worth saying that while we as humans die Companies and movements and economies don't have to because when they get to like late prime or aristocracy They can partner or in innovatively reinvent themselves, and that makes brands resilient One of the ways that we know that we're in adolescence is because we're getting red flags on the play that we're basically Playing it out so that we've now rubbed up against sufficiently policymakers and regulators and some other incumbent industries In cities and in places where we're working. That's a good thing from my perspective It means that we're maturing and that people are taking it seriously enough to begin to be concerned and engage So and lastly we have a very diverse Ecosystem that's emerging another way that we know that we're maturing as we see concierge services and Companies that are actually renting cars to people who want to be drivers and lift inside car who don't have their own cars Those secondary offerings are evidence of a more mature marketplace as well It's worth also just saying in brief that I'm hoping that in the next two days We'll also discuss the need for further diversification So that the collaborative economy isn't really something that looks like sharing for rich people But we have the ability to more broadly Provide access to many different services And I didn't even say Kansas Jeannie's definitely out of the bottle. No way we're going back Brands can play very big here one quick thing that I also and you'll be hearing from my colleagues shortly One of the big questions that I've been focused on in the last couple of years is the relationship between value capture and value Sorry value created and value captured Who creates the value and who gets to reap the rewards and is that equitable? Bless you. So if we have a relationship where we basically have For example, quirky, which is a crowd sourced inventions platform where they create consumer products and GE One of the generals that's working hard to make it into this new era Form a partnership and GE actually decided a couple of years ago. They're focused on B2B only not consumer so essentially they Were convinced by this very dynamic founder of quirky to give them To give quirky give in quotes the patent portfolio and put it on quirky's platform to make it available To its global community to actually use the technology that GE had invested in for other things Directed to consumer products. Well, the first thing that's launching this week is the arrows air conditioner Dr. Garthin Leslie was the is a retired 68 year old employee of the Department of Energy in the United States and He really hated these big ugly air conditioners in New York that turn out to use 20% of the electricity So together they created this new device. It's it's launching. It's great. It's good for both companies Certainly, but I asked the question. Well, what is dr. Leslie get and it turns out that with this model dr. Leslie gets $700,000 check As a as a compensation for the invention So we're looking at and we're exploring many different models for how we share the value and value doesn't only mean money But I welcome all those conversations in the next couple of days Please bite off more than you can chew and then yep chew like hell Welcome to San Francisco and thank you very much