<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Workflows
Although it would be possible to write analysis scripts using just Nipype Interfaces, and this may provide some advantages over directly making command-line calls, the main benefits of Nipype will come by creating workflows.
A workflow controls the setup and the execution of individual interfaces. Let's assume you want to run multiple interfaces in a specific order, where some have to wait for others to finish while others can be executed in parallel. The nice thing about a nipype workflow is, that the workflow  will take care of input and output of each interface and arrange the execution of each interface in the most efficient way.
A workflow therefore consists of multiple Nodes, each representing a specific Interface and directed connection between those nodes. Those connections specify which output of which node should be used as an input for another node. To better understand why this is so great, let's look at an example.
Preparation
Before we can start, let's first load some helper functions
Step1:  Example 1 - Command-line execution
Let's take a look at a small preprocessing analysis where we would like to perform the following steps of processing
Step2:  This is simple and straightforward. We can see that this does exactly what we wanted by plotting the four steps of processing.
Step3:  Example 2 - Interface execution
Now let's see what this would look like if we used Nipype, but only the Interfaces functionality. It's simple enough to write a basic procedural script, this time in Python, to do the same thing as above
Step4:  This is more verbose, although it does have its advantages. There's the automated input validation we saw previously, some of the options are named more meaningfully, and you don't need to remember, for example, that fslmaths' smoothing kernel is set in sigma instead of FWHM -- Nipype does that conversion behind the scenes.
Can't we optimize that a bit?
As we can see above, the inputs for the mask routine in_file and mask_file are actually the output of skullstrip and smooth. We therefore somehow want to connect them. This can be accomplisehd by saving the executed routines under a given object and than using the output of those objects as input for other routines.
Step5:  Here we didn't need to name the intermediate files; Nipype did that behind the scenes, and then we passed the result object (which knows those names) onto the next step in the processing stream. This is somewhat more concise than the example above, but it's still a procedural script. And the dependency relationship between the stages of processing is not particularly obvious. To address these issues, and to provide solutions to problems we might not know we have yet, Nipype offers Workflows.
Example 3 - Workflow execution
What we've implicitly done above is to encode our processing stream as a directed acyclic graphs
Step6:  This looks mostly similar to what we did above, but we've left out the two crucial inputs to the ApplyMask step. We'll set those up by defining a Workflow object and then making connections among the Nodes.
Step7:  The Workflow object has a method called connect that is going to do most of the work here. This routine also checks if inputs and outputs are actually provided by the nodes that are being connected.
There are two different ways to call connect
Step8:  Now the workflow is complete!
Above, we mentioned that the workflow can be thought of as a directed acyclic graph. In fact, that's literally how it's represented behind the scenes, and we can use that to explore the workflow visually
Step9:  This representation makes the dependency structure of the workflow obvious. (By the way, the names of the nodes in this graph are the names we gave our Node objects above, so pick something meaningful for those!)
Certain graph types also allow you to further inspect the individual connections between the nodes. For example
Step10:  Here you see very clearly, that the output mask_file of the skullstrip node is used as the input mask_file of the mask node. For more information on graph visualization, see the Graph Visualization section.
But let's come back to our example. At this point, all we've done is define the workflow. We haven't executed any code yet. Much like Interface objects, the Workflow object has a run method that we can call so that it executes. Let's do that and then examine the results.
Step11:  The specification of base_dir is very important (and is why we needed to use absolute paths above), because otherwise all the outputs would be saved somewhere in the temporary files. Unlike interfaces, which by default spit out results to the local directry, the Workflow engine executes things off in its own directory hierarchy.
Let's take a look at the resulting images to convince ourselves we've done the same thing as before
Step12:  Perfet!
Let's also have a closer look at the working directory
Step13:  As you can see, the name of the working directory is the name we gave the workflow base_dir. And the name of the folder within is the name of the workflow object smoothflow. Each node of the workflow has its' own subfolder in the smoothflow folder. And each of those subfolders contains the output of the node as well as some additional files.
A workflow inside a workflow
When you start writing full-fledged analysis workflows, things can get quite complicated. Some aspects of neuroimaging analysis can be thought of as a coherent step at a level more abstract than the execution of a single command line binary. For instance, in the standard FEAT script in FSL, several calls are made in the process of using susan to perform nonlinear smoothing on an image. In Nipype, you can write nested workflows, where a sub-workflow can take the place of a Node in a given script.
Let's use the prepackaged susan workflow that ships with Nipype to replace our Gaussian filtering node and demonstrate how this works.
Step14:  Calling this function will return a pre-written Workflow object
Step15:  Let's display the graph to see what happens here.
Step16:  We see that the workflow has an inputnode and an outputnode. While not strictly necessary, this is standard practice for workflows (especially those that are intended to be used as nested workflows in the context of a longer analysis graph) and makes it more clear how to connect inputs and outputs from this workflow.
Let's take a look at what those inputs and outputs are. Like Nodes, Workflows have inputs and outputs attributes that take a second sub-attribute corresponding to the specific node we want to make connections to.
Step17:  Note that inputnode and outputnode are just conventions, and the Workflow object exposes connections to all of its component nodes
Step18:  Let's see how we would write a new workflow that uses this nested smoothing step.
The susan workflow actually expects to receive and output a list of files (it's intended to be executed on each of several runs of fMRI data). We'll cover exactly how that works in later tutorials, but for the moment we need to add an additional Function node to deal with the fact that susan is outputting a list. We can use a simple lambda function to do this
Step19:  Now let's create a new workflow susanflow that contains the susan workflow as a sub-node. To be sure, let's also recreate the skullstrip and the mask node from the examples above.
Step20:  First, let's see what this new processing graph looks like.
Step21:  We can see how there is a nested smoothing workflow (blue) in the place of our previous smooth node. This provides a very detailed view, but what if you just wanted to give a higher-level summary of the processing steps? After all, that is the purpose of encapsulating smaller streams in a nested workflow. That, fortunately, is an option when writing out the graph
Step22:  That's much more managable. Now let's execute the workflow
Step23:  As a final step, let's look at the input and the output. It's exactly what we wanted.
Step24:  So, why are workflows so great?
So far, we've seen that you can build up rather complex analysis workflows. But at the moment, it's not been made clear why this is worth the extra trouble from writing a simple procedural script. To demonstrate the first added benefit of the Nipype, let's just rerun the susanflow workflow from above and measure the execution times.
Step25:  That happened quickly! Workflows (actually this is handled by the Node code) are smart, and know if their inputs have changed from the last time they are run. If they have not, they don't recompute; they just turn around and pass out the resulting files from the previous run. This is done on a node-by-node basis, also.
Let's go back to the first workflow example. What happened if we just tweak one thing
<ASSISTANT_TASK:>
Python Code:
%pylab inline
import nibabel as nb
# Let's create a short helper function to plot 3D NIfTI images
def plot_slice(fname):
    # Load the image
    img = nb.load(fname)
    data = img.get_data()
    # Cut in the middle of the brain
    cut = int(data.shape[-1]/2) + 10
    # Plot the data
    imshow(np.rot90(data[..., cut]), cmap="gray")
    gca().set_axis_off()
Explanation: Workflows
Although it would be possible to write analysis scripts using just Nipype Interfaces, and this may provide some advantages over directly making command-line calls, the main benefits of Nipype will come by creating workflows.
A workflow controls the setup and the execution of individual interfaces. Let's assume you want to run multiple interfaces in a specific order, where some have to wait for others to finish while others can be executed in parallel. The nice thing about a nipype workflow is, that the workflow  will take care of input and output of each interface and arrange the execution of each interface in the most efficient way.
A workflow therefore consists of multiple Nodes, each representing a specific Interface and directed connection between those nodes. Those connections specify which output of which node should be used as an input for another node. To better understand why this is so great, let's look at an example.
Preparation
Before we can start, let's first load some helper functions:
End of explanation
%%bash 
ANAT_NAME=sub-02_ses-test_T1w
ANAT=/data/ds000114/sub-02/ses-test/anat/${ANAT_NAME}
bet ${ANAT} /output/${ANAT_NAME}_brain -m -f 0.3
fslmaths ${ANAT} -s 2 /output/${ANAT_NAME}_smooth
fslmaths /output/${ANAT_NAME}_smooth -mas /output/${ANAT_NAME}_brain_mask /output/${ANAT_NAME}_smooth_mask
Explanation: Example 1 - Command-line execution
Let's take a look at a small preprocessing analysis where we would like to perform the following steps of processing:
- Skullstrip an image to obtain a mask
- Smooth the original image
- Mask the smoothed image
This could all very well be done with the following shell script:
End of explanation
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["T1w", "T1w_smooth",
                         "T1w_brain_mask", "T1w_smooth_mask"]):
    f.add_subplot(1, 4, i + 1)
    if i == 0:
        plot_slice("/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_%s.nii.gz" % img)
    else:
        plot_slice("/output/sub-02_ses-test_%s.nii.gz" % img)
    plt.title(img)
Explanation: This is simple and straightforward. We can see that this does exactly what we wanted by plotting the four steps of processing.
End of explanation
from nipype.interfaces import fsl
# Skullstrip process
skullstrip = fsl.BET(
    in_file="/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz",
    out_file="/output/sub-02_T1w_brain.nii.gz",
    mask=True)
skullstrip.run()
# Smoothing process
smooth = fsl.IsotropicSmooth(
    in_file="/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz",
    out_file="/output/sub-02_T1w_smooth.nii.gz",
    fwhm=4)
smooth.run()
# Masking process
mask = fsl.ApplyMask(
    in_file="/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz",
    out_file="/output/sub-02_T1w_smooth_mask.nii.gz",
    mask_file="/output/sub-02_T1w_brain_mask.nii.gz")
mask.run()
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["T1w", "T1w_smooth",
                         "T1w_brain_mask", "T1w_smooth_mask"]):
    f.add_subplot(1, 4, i + 1)
    if i == 0:
        plot_slice("/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_%s.nii.gz" % img)
    else:
        plot_slice("/output/sub-02_%s.nii.gz" % img)
    plt.title(img)
Explanation: Example 2 - Interface execution
Now let's see what this would look like if we used Nipype, but only the Interfaces functionality. It's simple enough to write a basic procedural script, this time in Python, to do the same thing as above:
End of explanation
from nipype.interfaces import fsl
# Skullstrip process
skullstrip = fsl.BET(
    in_file="/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz", mask=True)
bet_result = skullstrip.run()  # skullstrip object
# Smooth process
smooth = fsl.IsotropicSmooth(
    in_file="/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz", fwhm=4)
smooth_result = smooth.run()  # smooth object
# Mask process
mask = fsl.ApplyMask(in_file=smooth_result.outputs.out_file,
                     mask_file=bet_result.outputs.mask_file)
mask_result = mask.run()
f = plt.figure(figsize=(12, 4))
for i, img in enumerate([skullstrip.inputs.in_file, smooth_result.outputs.out_file,
                         bet_result.outputs.mask_file, mask_result.outputs.out_file]):
    f.add_subplot(1, 4, i + 1)
    plot_slice(img)
    plt.title(img.split('/')[-1].split('.')[0].split('test_')[-1])
Explanation: This is more verbose, although it does have its advantages. There's the automated input validation we saw previously, some of the options are named more meaningfully, and you don't need to remember, for example, that fslmaths' smoothing kernel is set in sigma instead of FWHM -- Nipype does that conversion behind the scenes.
Can't we optimize that a bit?
As we can see above, the inputs for the mask routine in_file and mask_file are actually the output of skullstrip and smooth. We therefore somehow want to connect them. This can be accomplisehd by saving the executed routines under a given object and than using the output of those objects as input for other routines.
End of explanation
# Import Node and Workflow object and FSL interface
from nipype import Node, Workflow
from nipype.interfaces import fsl
# For reasons that will later become clear, it's important to
# pass filenames to Nodes as absolute paths
from os.path import abspath
in_file = abspath("/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz")
# Skullstrip process
skullstrip = Node(fsl.BET(in_file=in_file, mask=True), name="skullstrip")
# Smooth process
smooth = Node(fsl.IsotropicSmooth(in_file=in_file, fwhm=4), name="smooth")
# Mask process
mask = Node(fsl.ApplyMask(), name="mask")
Explanation: Here we didn't need to name the intermediate files; Nipype did that behind the scenes, and then we passed the result object (which knows those names) onto the next step in the processing stream. This is somewhat more concise than the example above, but it's still a procedural script. And the dependency relationship between the stages of processing is not particularly obvious. To address these issues, and to provide solutions to problems we might not know we have yet, Nipype offers Workflows.
Example 3 - Workflow execution
What we've implicitly done above is to encode our processing stream as a directed acyclic graphs: each stage of processing is a node in this graph, and some nodes are unidirectionally dependent on others. In this case there is one input file and several output files, but there are no cycles -- there's a clear line of directionality to the processing. What the Node and Workflow classes do is make these relationships more explicit.
The basic architecture is that the Node provides a light wrapper around an Interface. It exposes the inputs and outputs of the Interface as its own, but it adds some additional functionality that allows you to connect Nodes into a Workflow.
Let's rewrite the above script with these tools:
End of explanation
# Initiation of a workflow
wf = Workflow(name="smoothflow", base_dir="/output/working_dir")
Explanation: This looks mostly similar to what we did above, but we've left out the two crucial inputs to the ApplyMask step. We'll set those up by defining a Workflow object and then making connections among the Nodes.
End of explanation
# First the "simple", but more restricted method
wf.connect(skullstrip, "mask_file", mask, "mask_file")
# Now the more complicated method
wf.connect([(smooth, mask, [("out_file", "in_file")])])
Explanation: The Workflow object has a method called connect that is going to do most of the work here. This routine also checks if inputs and outputs are actually provided by the nodes that are being connected.
There are two different ways to call connect:
connect(source, "source_output", dest, "dest_input")
connect([(source, dest, [("source_output1", "dest_input1"),
                         ("source_output2", "dest_input2")
                         ])
         ])
With the first approach you can establish one connection at a time. With the second you can establish multiple connects between two nodes at once. In either case, you're providing it with four pieces of information to define the connection:
The source node object
The name of the output field from the source node
The destination node object
The name of the input field from the destination node
We'll illustrate each method in the following cell:
End of explanation
wf.write_graph("workflow_graph.dot")
from IPython.display import Image
Image(filename="/output/working_dir/smoothflow/workflow_graph.dot.png")
Explanation: Now the workflow is complete!
Above, we mentioned that the workflow can be thought of as a directed acyclic graph. In fact, that's literally how it's represented behind the scenes, and we can use that to explore the workflow visually:
End of explanation
wf.write_graph(graph2use='flat')
from IPython.display import Image
Image(filename="/output/working_dir/smoothflow/graph_detailed.dot.png")
Explanation: This representation makes the dependency structure of the workflow obvious. (By the way, the names of the nodes in this graph are the names we gave our Node objects above, so pick something meaningful for those!)
Certain graph types also allow you to further inspect the individual connections between the nodes. For example:
End of explanation
# Specify the base directory for the working directory
wf.base_dir = "/output/working_dir"
# Execute the workflow
wf.run()
Explanation: Here you see very clearly, that the output mask_file of the skullstrip node is used as the input mask_file of the mask node. For more information on graph visualization, see the Graph Visualization section.
But let's come back to our example. At this point, all we've done is define the workflow. We haven't executed any code yet. Much like Interface objects, the Workflow object has a run method that we can call so that it executes. Let's do that and then examine the results.
End of explanation
f = plt.figure(figsize=(12, 4))
for i, img in enumerate(["/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz",
                         "/output/working_dir/smoothflow/smooth/sub-02_ses-test_T1w_smooth.nii.gz",
                         "/output/working_dir/smoothflow/skullstrip/sub-02_ses-test_T1w_brain_mask.nii.gz",
                         "/output/working_dir/smoothflow/mask/sub-02_ses-test_T1w_smooth_masked.nii.gz"]):
    f.add_subplot(1, 4, i + 1)
    plot_slice(img)
Explanation: The specification of base_dir is very important (and is why we needed to use absolute paths above), because otherwise all the outputs would be saved somewhere in the temporary files. Unlike interfaces, which by default spit out results to the local directry, the Workflow engine executes things off in its own directory hierarchy.
Let's take a look at the resulting images to convince ourselves we've done the same thing as before:
End of explanation
!tree /output/working_dir/smoothflow/ -I '*js|*json|*html|*pklz|_report'
Explanation: Perfet!
Let's also have a closer look at the working directory:
End of explanation
from nipype.workflows.fmri.fsl import create_susan_smooth
Explanation: As you can see, the name of the working directory is the name we gave the workflow base_dir. And the name of the folder within is the name of the workflow object smoothflow. Each node of the workflow has its' own subfolder in the smoothflow folder. And each of those subfolders contains the output of the node as well as some additional files.
A workflow inside a workflow
When you start writing full-fledged analysis workflows, things can get quite complicated. Some aspects of neuroimaging analysis can be thought of as a coherent step at a level more abstract than the execution of a single command line binary. For instance, in the standard FEAT script in FSL, several calls are made in the process of using susan to perform nonlinear smoothing on an image. In Nipype, you can write nested workflows, where a sub-workflow can take the place of a Node in a given script.
Let's use the prepackaged susan workflow that ships with Nipype to replace our Gaussian filtering node and demonstrate how this works.
End of explanation
susan = create_susan_smooth(separate_masks=False)
Explanation: Calling this function will return a pre-written Workflow object:
End of explanation
susan.write_graph("susan_workflow.dot")
from IPython.display import Image
Image(filename="susan_workflow.dot.png")
Explanation: Let's display the graph to see what happens here.
End of explanation
print("Inputs:\n", susan.inputs.inputnode)
print("Outputs:\n", susan.outputs.outputnode)
Explanation: We see that the workflow has an inputnode and an outputnode. While not strictly necessary, this is standard practice for workflows (especially those that are intended to be used as nested workflows in the context of a longer analysis graph) and makes it more clear how to connect inputs and outputs from this workflow.
Let's take a look at what those inputs and outputs are. Like Nodes, Workflows have inputs and outputs attributes that take a second sub-attribute corresponding to the specific node we want to make connections to.
End of explanation
susan.inputs
Explanation: Note that inputnode and outputnode are just conventions, and the Workflow object exposes connections to all of its component nodes:
End of explanation
from nipype import Function
extract_func = lambda list_out: list_out[0]
list_extract = Node(Function(input_names=["list_out"],
                             output_names=["out_file"],
                             function=extract_func),
                    name="list_extract")
Explanation: Let's see how we would write a new workflow that uses this nested smoothing step.
The susan workflow actually expects to receive and output a list of files (it's intended to be executed on each of several runs of fMRI data). We'll cover exactly how that works in later tutorials, but for the moment we need to add an additional Function node to deal with the fact that susan is outputting a list. We can use a simple lambda function to do this:
End of explanation
# Initiate workflow with name and base directory
wf2 = Workflow(name="susanflow", base_dir="/output/working_dir")
# Create new skullstrip and mask nodes
skullstrip2 = Node(fsl.BET(in_file=in_file, mask=True), name="skullstrip")
mask2 = Node(fsl.ApplyMask(), name="mask")
# Connect the nodes to each other and to the susan workflow
wf2.connect([(skullstrip2, mask2, [("mask_file", "mask_file")]),
             (skullstrip2, susan, [("mask_file", "inputnode.mask_file")]),
             (susan, list_extract, [("outputnode.smoothed_files",
                                     "list_out")]),
             (list_extract, mask2, [("out_file", "in_file")])
             ])
# Specify the remaining input variables for the susan workflow
susan.inputs.inputnode.in_files = abspath(
    "/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz")
susan.inputs.inputnode.fwhm = 4
Explanation: Now let's create a new workflow susanflow that contains the susan workflow as a sub-node. To be sure, let's also recreate the skullstrip and the mask node from the examples above.
End of explanation
wf2.write_graph(dotfilename='/output/working_dir/full_susanflow.dot', graph2use='colored')
from IPython.display import Image
Image(filename="/output/working_dir/full_susanflow.dot.png")
Explanation: First, let's see what this new processing graph looks like.
End of explanation
wf2.write_graph(dotfilename='/output/working_dir/full_susanflow_toplevel.dot', graph2use='orig')
from IPython.display import Image
Image(filename="/output/working_dir/full_susanflow_toplevel.dot.png")
Explanation: We can see how there is a nested smoothing workflow (blue) in the place of our previous smooth node. This provides a very detailed view, but what if you just wanted to give a higher-level summary of the processing steps? After all, that is the purpose of encapsulating smaller streams in a nested workflow. That, fortunately, is an option when writing out the graph:
End of explanation
wf2.run()
Explanation: That's much more managable. Now let's execute the workflow
End of explanation
f = plt.figure(figsize=(12, 4))
for i, e in enumerate([["/data/ds000114/sub-02/ses-test/anat/sub-02_ses-test_T1w.nii.gz", 'input'],
                       ["/output/working_dir//susanflow/mask/sub-02_ses-test_T1w_smooth_masked.nii.gz", 
                        'output']]):
    f.add_subplot(1, 2, i + 1)
    plot_slice(e[0])
    plt.title(e[1])
Explanation: As a final step, let's look at the input and the output. It's exactly what we wanted.
End of explanation
%time
wf2.run()
Explanation: So, why are workflows so great?
So far, we've seen that you can build up rather complex analysis workflows. But at the moment, it's not been made clear why this is worth the extra trouble from writing a simple procedural script. To demonstrate the first added benefit of the Nipype, let's just rerun the susanflow workflow from above and measure the execution times.
End of explanation
wf.inputs.smooth.fwhm = 1
wf.run()
Explanation: That happened quickly! Workflows (actually this is handled by the Node code) are smart, and know if their inputs have changed from the last time they are run. If they have not, they don't recompute; they just turn around and pass out the resulting files from the previous run. This is done on a node-by-node basis, also.
Let's go back to the first workflow example. What happened if we just tweak one thing:
End of explanation
<END_TASK>