<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Hands-on 1
Step1:  Preprocessing Workflow Structure
So let's get our hands dirty. First things first, it's always good to know which interfaces you want to use in your workflow and in which order you want to execute them. For the preprocessing workflow, I recommend that we use the following nodes
Step2:  Note
Step3:  Gunzip
I've already created the Gunzip node as a template for the other nodes. Also, we've specified an in_file here so that we can directly test the nodes without worrying about the Input/Output data stream to the workflow. This will be taken care of in a later section.
Step4:  Drop Dummy Scans
The functional images of this dataset were recorded with 4 dummy scans at the beginning (see the corresponding publication). But those dummy scans were not yet taken out from the functional images.
To better illustrate this, let's plot the time course of a random voxel of the just defined func_file
Step5:  In the figure above, we see that at the very beginning there are extreme values, which hint to the fact that steady state wasn't reached yet. Therefore, we want to exclude the dummy scans from the original data. This can be achieved with FSL's ExtractROI.
Step6:  This ExtractROI node can now be connected to the gunzip_func node from above. To do this, we use the following command
Step7:  Slice Time Correction
Now to the next step. Let's us SPM's SliceTiming to correct for slice wise acquisition of the volumes. As a reminder, the tutorial dataset was recorded...
- with a time repetition (TR) of 2.5 seconds
- with 30 slices per volume
- in an interleaved fashion, i.e. slice order is [1, 3, 5, 7, ..., 2, 4, 6, ..., 30]
- with a time acquisition (TA) of 2.4167 seconds, i.e. TR-(TR/num_slices)
Step8:  Now the next step is to connect the SliceTiming node to the rest of the workflow, i.e. the ExtractROI node.
Step9:  Motion Correction
To correct for motion in the scanner, we will be using FSL's MCFLIRT.
Step10:  Connect the MCFLIRT node to the rest of the workflow.
Step11:  Artifact Detection
We will use the really cool and useful ArtifactDetection tool from Nipype to detect motion and intensity outliers in the functional images. The interface is initiated as follows
Step12:  The parameters above mean the following
Step13:  Segmentation of anatomical image
Now let's work on the anatomical image. In particular, let's use SPM's NewSegment to create probability maps for the gray matter, white matter tissue and CSF.
Step14:  We will again be using a Gunzip node to unzip the anatomical image that we then want to use as input to the segmentation node. We again also need to specify the anatomical image that we want to use in this case. As before, this will later also be handled directly by the Input/Output stream.
Step15:  Now we can connect the NewSegment node to the rest of the workflow.
Step16:  Compute Coregistration Matrix
As a next step, we will make sure that the functional images are coregistered to the anatomical image. For this, we will use FSL's FLIRT function. As we just created a white matter probability map, we can use this together with the Boundary-Based Registration (BBR) cost function to optimize the image coregistration. As some helpful notes...
- use a degree of freedom of 6
- specify the cost function as bbr
- use the schedule='/usr/share/fsl/5.0/etc/flirtsch/bbr.sch'
Step17:  As mentioned above, the bbr routine can use the subject-specific white matter probability map to guide the coregistration. But for this, we need to create a binary mask out of the WM probability map. This can easily be done by FSL's Threshold interface.
Step18:  Now, to select the WM probability map that the NewSegment node created, we need some helper function. Because the output field partial_volume_files form the segmentation node, will give us a list of files, i.e. [[GM_prob], [WM_prob], [], [], [], []]. Therefore, using the following function, we can select only the last element of this list.
Step19:  Now we can just connect this Threshold node to the coregistration node from above.
Step20:  Apply Coregistration Matrix to functional image
Now that we know the coregistration matrix to correctly overlay the functional mean image on the subject-specific anatomy, we need to apply to coregistration to the whole time series. This can be achieved with FSL's FLIRT as follows
Step21:  <span style="color
Step22:  Smoothing
Next step is image smoothing. The most simple way to do this is to use FSL's or SPM's Smooth function. But for learning purposes, let's use FSL's SUSAN workflow as it is implemented in Nipype. Note that this time, we are importing a workflow instead of an interface.
Step23:  If you type create_susan_smooth? you can see how to specify the input variables to the susan workflow. In particular, they are...
- fwhm
Step24:  Create Binary Mask
There are many possible approaches on how you can mask your functional images. One of them is not at all, one is with a simple brain mask and one that only considers certain kind of brain tissue, e.g. gray matter.
For the current example, we want to create a dilated gray matter mask. For this purpose we need to
Step25:  The second and third step can luckily be done with just one node. We can take almost the same Threshold node as above. We just need to add another additional argument
Step26:  Now we can connect the resample and the gray matter mask node to the segmentation node and each other.
Step27:  This should do the trick.
Apply the binary mask
Now we can connect this dilated gray matter mask to the susan node, as well as actually applying this to the resulting smoothed images.
Step28:  To apply the mask to the smoothed functional images, we will use FSL's ApplyMask interface.
Step29:  Important
Step30:  Remove linear trends in functional images
Last but not least. Let's use Nipype's TSNR module to remove linear and quadratic trends in the functionally smoothed images. For this, you only have to specify the regress_poly parameter in the node initiation.
Step31:  Datainput with SelectFiles and iterables
This is all nice and well. But so far we still had to specify the input values for gunzip_anat and gunzip_func ourselves. How can we scale this up to multiple subjects and/or multiple functional images and make the workflow take the input directly from the BIDS dataset?
For this, we need SelectFiles and iterables! It's rather simple, specify a template and fill-up the placeholder variables.
Step32:  Now we can specify over which subjects the workflow should iterate. To test the workflow, let's still just look at subject 7.
Step33:  Visualize the workflow
Now that we're done. Let's look at the workflow that we just created.
Step34:  Run the Workflow
Now we are ready to run the workflow! Be careful about the n_procs parameter if you run a workflow in 'MultiProc' mode. n_procs specifies the number of jobs/cores your computer will use to run the workflow. If this number is too high your computer will try to execute too many things at once and will most likely crash.
Note
Step35:  Inspect output
What did we actually do? Let's look at all the data that was created.
Step36:  But what did we do specifically? Well, let's investigate.
Motion Correction and Artifact Detection
How much did the subject move in the scanner and where there any outliers in the functional images?
Step37:  The motion parameters seems to look ok. What about the detection of artifacts?
Step38:  Which volumes are problematic?
Step39:  Masks and Probability maps
Let's see what all the masks and probability maps look like. For this, we will use nilearn's plot_anat function.
Step40:  First, let's look at the tissue probability maps.
Step41:  And how does the gray matter mask look like that we used on the functional images?
Step42:  Functional Image transformations
Let's also investigate the transformation that we applied to the functional images.
Step43:  That's all nice and beautiful, but what did smoothing and detrending actually do to the data?
Step44:  Data output with DataSink
The results look fine, but we don't need all those temporary files. So let's use Datasink to keep only those files that we actually need for the 1st and 2nd level analysis.
Step45:  Now the next step is to specify all the output that we want to keep in our output folder output. Make sure to keep
Step46:  Run the workflow
After adding the datasink folder, let's run the preprocessing workflow again.
Step47:  Let's look now at the output of this datasink folder.
Step48:  Much better! But we're still not there yet. There are many unnecessary file specifiers that we can get rid off. To do so, we can use DataSink's substitutions parameter. For this, we create a list of tuples
Step49:  Before we run the preprocessing workflow again, let's first delete the current output folder
Step50:  Run Preprocessing workflow on 6 right-handed subjects
Perfect! Now let's run the whole workflow for right-handed subjects. For this, you just need to change the subject_list variable and run again the places where this variable is used (i.e. sf.iterables and in DataSink substitutions.
Step51:  Now we can run the workflow again, this time for all right handed subjects in parallel.
<ASSISTANT_TASK:>
Python Code:
%%bash
datalad get -J 4 -d /data/ds000114 \
    /data/ds000114/sub-0[234789]/ses-test/anat/sub-0[234789]_ses-test_T1w.nii.gz \
    /data/ds000114/sub-0[234789]/ses-test/func/*fingerfootlips*
Explanation: Hands-on 1: How to create a fMRI preprocessing workflow
The purpose of this section is that you set-up a complete fMRI analysis workflow yourself. So that in the end, you are able to perform the analysis from A-Z, i.e. from preprocessing to group analysis. This section will cover the preprocessing part, and the section Hands-on 2: Analysis will handle the analysis part.
We will use this opportunity to show you some nice additional interfaces/nodes that might not be relevant to your usual analysis. But it's always nice to know that they exist. And hopefully, this will encourage you to investigate all other interfaces that Nipype can bring to the tip of your finger.
Preparation
Before we can start with anything we first need to download the data. For this hands-on, we will only use the right-handed subjects 2-4 and 7-9. This can be done very quickly with the following datalad command.
Note: This might take a while, as datalad needs to download ~200MB of data
End of explanation
from nilearn import plotting
%matplotlib inline
# Get the Node and Workflow object
from nipype import Node, Workflow
# Specify which SPM to use
from nipype.interfaces.matlab import MatlabCommand
MatlabCommand.set_default_paths('/opt/spm12-r7219/spm12_mcr/spm12')
Explanation: Preprocessing Workflow Structure
So let's get our hands dirty. First things first, it's always good to know which interfaces you want to use in your workflow and in which order you want to execute them. For the preprocessing workflow, I recommend that we use the following nodes:
 1. Gunzip (Nipype)
 2. Drop Dummy Scans (FSL)
 3. Slice Time Correction (SPM)
 4. Motion Correction (SPM)
 5. Artifact Detection
 6. Segmentation (SPM)
 7. Coregistration (FSL)
 8. Smoothing (FSL)
 9. Apply Binary Mask (FSL)
10. Remove Linear Trends (Nipype)
Note: This workflow might be overkill concerning data manipulation, but it hopefully serves as a good Nipype exercise.
Imports
It's always best to have all relevant module imports at the beginning of your script. So let's import what we most certainly need.
End of explanation
# Create the workflow here
# Hint: use 'base_dir' to specify where to store the working directory
preproc = Workflow(name='work_preproc', base_dir='/output/')
Explanation: Note: Ideally you would also put the imports of all the interfaces that you use here at the top. But as we will develop the workflow step by step, we can also import the relevant modules as we go.
Create Nodes and Workflow connections
Let's create all the nodes that we need! Make sure to specify all relevant inputs and keep in mind which ones you later on need to connect in your pipeline.
Workflow
We recommend to create the workflow and establish all its connections at a later place in your script. This helps to have everything nicely together. But for this hands-on example, it makes sense to establish the connections between the nodes as we go.
And for this, we first need to create a workflow:
End of explanation
from nipype.algorithms.misc import Gunzip
# Specify example input file
func_file = '/data/ds000114/sub-07/ses-test/func/sub-07_ses-test_task-fingerfootlips_bold.nii.gz'
# Initiate Gunzip node
gunzip_func = Node(Gunzip(in_file=func_file), name='gunzip_func')
Explanation: Gunzip
I've already created the Gunzip node as a template for the other nodes. Also, we've specified an in_file here so that we can directly test the nodes without worrying about the Input/Output data stream to the workflow. This will be taken care of in a later section.
End of explanation
import nibabel as nb
%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(nb.load(func_file).get_fdata()[32, 32, 15, :]);
Explanation: Drop Dummy Scans
The functional images of this dataset were recorded with 4 dummy scans at the beginning (see the corresponding publication). But those dummy scans were not yet taken out from the functional images.
To better illustrate this, let's plot the time course of a random voxel of the just defined func_file:
End of explanation
from nipype.interfaces.fsl import ExtractROI
extract = Node(ExtractROI(t_min=4, t_size=-1, output_type='NIFTI'),
               name="extract")
Explanation: In the figure above, we see that at the very beginning there are extreme values, which hint to the fact that steady state wasn't reached yet. Therefore, we want to exclude the dummy scans from the original data. This can be achieved with FSL's ExtractROI.
End of explanation
preproc.connect([(gunzip_func, extract, [('out_file', 'in_file')])])
Explanation: This ExtractROI node can now be connected to the gunzip_func node from above. To do this, we use the following command:
End of explanation
from nipype.interfaces.spm import SliceTiming
slice_order = list(range(1, 31, 2)) + list(range(2, 31, 2))
print(slice_order)
# Initiate SliceTiming node here
slicetime = Node(SliceTiming(num_slices=30,
                             ref_slice=15,
                             slice_order=slice_order,
                             time_repetition=2.5,
                             time_acquisition=2.5-(2.5/30)),
                 name='slicetime')
Explanation: Slice Time Correction
Now to the next step. Let's us SPM's SliceTiming to correct for slice wise acquisition of the volumes. As a reminder, the tutorial dataset was recorded...
- with a time repetition (TR) of 2.5 seconds
- with 30 slices per volume
- in an interleaved fashion, i.e. slice order is [1, 3, 5, 7, ..., 2, 4, 6, ..., 30]
- with a time acquisition (TA) of 2.4167 seconds, i.e. TR-(TR/num_slices)
End of explanation
# Connect SliceTiming node to the other nodes here
preproc.connect([(extract, slicetime, [('roi_file', 'in_files')])])
Explanation: Now the next step is to connect the SliceTiming node to the rest of the workflow, i.e. the ExtractROI node.
End of explanation
from nipype.interfaces.fsl import MCFLIRT
# Initate MCFLIRT node here
mcflirt = Node(MCFLIRT(mean_vol=True,
                       save_plots=True),
               name="mcflirt")
Explanation: Motion Correction
To correct for motion in the scanner, we will be using FSL's MCFLIRT.
End of explanation
# Connect MCFLIRT node to the other nodes here
preproc.connect([(slicetime, mcflirt, [('timecorrected_files', 'in_file')])])
Explanation: Connect the MCFLIRT node to the rest of the workflow.
End of explanation
from nipype.algorithms.rapidart import ArtifactDetect
art = Node(ArtifactDetect(norm_threshold=2,
                          zintensity_threshold=3,
                          mask_type='spm_global',
                          parameter_source='FSL',
                          use_differences=[True, False],
                          plot_type='svg'),
           name="art")
Explanation: Artifact Detection
We will use the really cool and useful ArtifactDetection tool from Nipype to detect motion and intensity outliers in the functional images. The interface is initiated as follows:
End of explanation
preproc.connect([(mcflirt, art, [('out_file', 'realigned_files'),
                                 ('par_file', 'realignment_parameters')])
                 ])
Explanation: The parameters above mean the following:
- norm_threshold - Threshold to use to detect motion-related outliers when composite motion is being used
- zintensity_threshold - Intensity Z-threshold use to detection images that deviate from the mean
- mask_type - Type of mask that should be used to mask the functional data. spm_global uses an spm_global like calculation to determine the brain mask
- parameter_source - Source of movement parameters
- use_differences - If you want to use differences between successive motion (first element) and intensity parameter (second element) estimates in order to determine outliers
And this is how you connect this node to the rest of the workflow:
End of explanation
from nipype.interfaces.spm import NewSegment
# Use the following tissue specification to get a GM and WM probability map
tpm_img ='/opt/spm12-r7219/spm12_mcr/spm12/tpm/TPM.nii'
tissue1 = ((tpm_img, 1), 1, (True,False), (False, False))
tissue2 = ((tpm_img, 2), 1, (True,False), (False, False))
tissue3 = ((tpm_img, 3), 2, (True,False), (False, False))
tissue4 = ((tpm_img, 4), 3, (False,False), (False, False))
tissue5 = ((tpm_img, 5), 4, (False,False), (False, False))
tissue6 = ((tpm_img, 6), 2, (False,False), (False, False))
tissues = [tissue1, tissue2, tissue3, tissue4, tissue5, tissue6]
# Initiate NewSegment node here
segment = Node(NewSegment(tissues=tissues), name='segment')
Explanation: Segmentation of anatomical image
Now let's work on the anatomical image. In particular, let's use SPM's NewSegment to create probability maps for the gray matter, white matter tissue and CSF.
End of explanation
# Specify example input file
anat_file = '/data/ds000114/sub-07/ses-test/anat/sub-07_ses-test_T1w.nii.gz'
# Initiate Gunzip node
gunzip_anat = Node(Gunzip(in_file=anat_file), name='gunzip_anat')
Explanation: We will again be using a Gunzip node to unzip the anatomical image that we then want to use as input to the segmentation node. We again also need to specify the anatomical image that we want to use in this case. As before, this will later also be handled directly by the Input/Output stream.
End of explanation
# Connect NewSegment node to the other nodes here
preproc.connect([(gunzip_anat, segment, [('out_file', 'channel_files')])])
Explanation: Now we can connect the NewSegment node to the rest of the workflow.
End of explanation
from nipype.interfaces.fsl import FLIRT
# Initiate FLIRT node here
coreg = Node(FLIRT(dof=6,
                   cost='bbr',
                   schedule='/usr/share/fsl/5.0/etc/flirtsch/bbr.sch',
                   output_type='NIFTI'),
             name="coreg")
# Connect FLIRT node to the other nodes here
preproc.connect([(gunzip_anat, coreg, [('out_file', 'reference')]),
                 (mcflirt, coreg, [('mean_img', 'in_file')])
                 ])
Explanation: Compute Coregistration Matrix
As a next step, we will make sure that the functional images are coregistered to the anatomical image. For this, we will use FSL's FLIRT function. As we just created a white matter probability map, we can use this together with the Boundary-Based Registration (BBR) cost function to optimize the image coregistration. As some helpful notes...
- use a degree of freedom of 6
- specify the cost function as bbr
- use the schedule='/usr/share/fsl/5.0/etc/flirtsch/bbr.sch'
End of explanation
from nipype.interfaces.fsl import Threshold
# Threshold - Threshold WM probability image
threshold_WM = Node(Threshold(thresh=0.5,
                              args='-bin',
                              output_type='NIFTI'),
                name="threshold_WM")
Explanation: As mentioned above, the bbr routine can use the subject-specific white matter probability map to guide the coregistration. But for this, we need to create a binary mask out of the WM probability map. This can easily be done by FSL's Threshold interface.
End of explanation
# Select WM segmentation file from segmentation output
def get_wm(files):
    return files[1][0]
# Connecting the segmentation node with the threshold node
preproc.connect([(segment, threshold_WM, [(('native_class_images', get_wm),
                                           'in_file')])])
Explanation: Now, to select the WM probability map that the NewSegment node created, we need some helper function. Because the output field partial_volume_files form the segmentation node, will give us a list of files, i.e. [[GM_prob], [WM_prob], [], [], [], []]. Therefore, using the following function, we can select only the last element of this list.
End of explanation
# Connect Threshold node to coregistration node above here
preproc.connect([(threshold_WM, coreg, [('out_file', 'wm_seg')])])
Explanation: Now we can just connect this Threshold node to the coregistration node from above.
End of explanation
# Specify the isometric voxel resolution you want after coregistration
desired_voxel_iso = 4
# Apply coregistration warp to functional images
applywarp = Node(FLIRT(interp='spline',
                       apply_isoxfm=desired_voxel_iso,
                       output_type='NIFTI'),
                 name="applywarp")
Explanation: Apply Coregistration Matrix to functional image
Now that we know the coregistration matrix to correctly overlay the functional mean image on the subject-specific anatomy, we need to apply to coregistration to the whole time series. This can be achieved with FSL's FLIRT as follows:
End of explanation
# Connecting the ApplyWarp node to all the other nodes
preproc.connect([(mcflirt, applywarp, [('out_file', 'in_file')]),
                 (coreg, applywarp, [('out_matrix_file', 'in_matrix_file')]),
                 (gunzip_anat, applywarp, [('out_file', 'reference')])
                 ])
Explanation: <span style="color:red">Important</span>: As you can see above, we also specified a variable desired_voxel_iso. This is very important at this stage, otherwise FLIRT will transform your functional images to a resolution of the anatomical image, which will dramatically increase the file size (e.g. to 1-10GB per file). If you don't want to change the voxel resolution, use the additional parameter no_resample=True. Important, for this to work, you still need to define apply_isoxfm.
End of explanation
from niflow.nipype1.workflows.fmri.fsl.preprocess import create_susan_smooth
Explanation: Smoothing
Next step is image smoothing. The most simple way to do this is to use FSL's or SPM's Smooth function. But for learning purposes, let's use FSL's SUSAN workflow as it is implemented in Nipype. Note that this time, we are importing a workflow instead of an interface.
End of explanation
# Initiate SUSAN workflow here
susan = create_susan_smooth(name='susan')
susan.inputs.inputnode.fwhm = 4
# Connect Threshold node to coregistration node above here
preproc.connect([(applywarp, susan, [('out_file', 'inputnode.in_files')])])
Explanation: If you type create_susan_smooth? you can see how to specify the input variables to the susan workflow. In particular, they are...
- fwhm: set this value to 4 (or whichever value you want)
- mask_file: will be created in a later step
- in_file: will be handled while connection to other nodes in the preproc workflow
End of explanation
from nipype.interfaces.fsl import FLIRT
# Initiate resample node
resample = Node(FLIRT(apply_isoxfm=desired_voxel_iso,
                      output_type='NIFTI'),
                name="resample")
Explanation: Create Binary Mask
There are many possible approaches on how you can mask your functional images. One of them is not at all, one is with a simple brain mask and one that only considers certain kind of brain tissue, e.g. gray matter.
For the current example, we want to create a dilated gray matter mask. For this purpose we need to:
1. Resample the gray matter probability map to the same resolution as the functional images
2. Threshold this resampled probability map at a specific value
3. Dilate this mask by some voxels to make the mask less conservative and more inclusive
The first step can be done in many ways (eg. using freesurfer's mri_convert, nibabel) but in our case, we will use FSL's FLIRT. The trick is to use the probability mask, as input file and a reference file.
End of explanation
from nipype.interfaces.fsl import Threshold
# Threshold - Threshold GM probability image
mask_GM = Node(Threshold(thresh=0.5,
                         args='-bin -dilF',
                         output_type='NIFTI'),
                name="mask_GM")
# Select GM segmentation file from segmentation output
def get_gm(files):
    return files[0][0]
Explanation: The second and third step can luckily be done with just one node. We can take almost the same Threshold node as above. We just need to add another additional argument: -dilF - which applies a maximum filtering of all voxels.
End of explanation
preproc.connect([(segment, resample, [(('native_class_images', get_gm), 'in_file'),
                                      (('native_class_images', get_gm), 'reference')
                                      ]),
                 (resample, mask_GM, [('out_file', 'in_file')])
                 ])
Explanation: Now we can connect the resample and the gray matter mask node to the segmentation node and each other.
End of explanation
# Connect gray matter Mask node to the susan workflow here
preproc.connect([(mask_GM, susan, [('out_file', 'inputnode.mask_file')])])
Explanation: This should do the trick.
Apply the binary mask
Now we can connect this dilated gray matter mask to the susan node, as well as actually applying this to the resulting smoothed images.
End of explanation
from nipype.interfaces.fsl import ApplyMask
Explanation: To apply the mask to the smoothed functional images, we will use FSL's ApplyMask interface.
End of explanation
from nipype import MapNode
# Initiate ApplyMask node here
mask_func = MapNode(ApplyMask(output_type='NIFTI'),
                    name="mask_func", 
                    iterfield=["in_file"])
# Connect smoothed susan output file to ApplyMask node here
preproc.connect([(susan, mask_func, [('outputnode.smoothed_files', 'in_file')]),
                 (mask_GM, mask_func, [('out_file', 'mask_file')])
                 ])
Explanation: Important: The susan workflow gives out a list of files, i.e. [smoothed_func.nii] instead of just the filename directly. If we would use a normal Node for ApplyMask this would lead to the following error:
TraitError: The 'in_file' trait of an ApplyMaskInput instance must be an existing file name, but a value of ['/output/work_preproc/susan/smooth/mapflow/_smooth0/asub-07_ses-test_task-fingerfootlips_bold_mcf_flirt_smooth.nii.gz'] &lt;class 'list'&gt; was specified.
To prevent this we will be using a MapNode and specify the in_file as it's iterfield. Like this, the node is capable to handle a list of inputs as it will know that it has to apply itself iteratively to the list of inputs.
End of explanation
from nipype.algorithms.confounds import TSNR
# Initiate TSNR node here
detrend = Node(TSNR(regress_poly=2), name="detrend")
# Connect the detrend node to the other nodes here
preproc.connect([(mask_func, detrend, [('out_file', 'in_file')])])
Explanation: Remove linear trends in functional images
Last but not least. Let's use Nipype's TSNR module to remove linear and quadratic trends in the functionally smoothed images. For this, you only have to specify the regress_poly parameter in the node initiation.
End of explanation
# Import the SelectFiles
from nipype import SelectFiles
# String template with {}-based strings
templates = {'anat': 'sub-{subject_id}/ses-{ses_id}/anat/'
                     'sub-{subject_id}_ses-test_T1w.nii.gz',
             'func': 'sub-{subject_id}/ses-{ses_id}/func/'
                     'sub-{subject_id}_ses-{ses_id}_task-{task_id}_bold.nii.gz'}
# Create SelectFiles node
sf = Node(SelectFiles(templates,
                      base_directory='/data/ds000114',
                      sort_filelist=True),
          name='selectfiles')
sf.inputs.ses_id='test'
sf.inputs.task_id='fingerfootlips'
Explanation: Datainput with SelectFiles and iterables
This is all nice and well. But so far we still had to specify the input values for gunzip_anat and gunzip_func ourselves. How can we scale this up to multiple subjects and/or multiple functional images and make the workflow take the input directly from the BIDS dataset?
For this, we need SelectFiles and iterables! It's rather simple, specify a template and fill-up the placeholder variables.
End of explanation
subject_list = ['07']
sf.iterables = [('subject_id', subject_list)]
# Connect SelectFiles node to the other nodes here
preproc.connect([(sf, gunzip_anat, [('anat', 'in_file')]),
                 (sf, gunzip_func, [('func', 'in_file')])])
Explanation: Now we can specify over which subjects the workflow should iterate. To test the workflow, let's still just look at subject 7.
End of explanation
# Create preproc output graph
preproc.write_graph(graph2use='colored', format='png', simple_form=True)
# Visualize the graph
from IPython.display import Image
Image(filename='/output/work_preproc/graph.png', width=750)
Explanation: Visualize the workflow
Now that we're done. Let's look at the workflow that we just created.
End of explanation
preproc.run('MultiProc', plugin_args={'n_procs': 4})
Explanation: Run the Workflow
Now we are ready to run the workflow! Be careful about the n_procs parameter if you run a workflow in 'MultiProc' mode. n_procs specifies the number of jobs/cores your computer will use to run the workflow. If this number is too high your computer will try to execute too many things at once and will most likely crash.
Note: If  you're using a Docker container and FLIRT fails to run without any good reason, you might need to change memory settings in the Docker preferences (6 GB should be enough for this workflow).
End of explanation
!tree /output/work_preproc/ -I '*js|*json|*pklz|_report|*dot|*html|*txt|*.m'
Explanation: Inspect output
What did we actually do? Let's look at all the data that was created.
End of explanation
%matplotlib inline
# Plot the motion paramters
import numpy as np
import matplotlib.pyplot as plt
par = np.loadtxt('/output/work_preproc/_subject_id_07/mcflirt/'
                 'asub-07_ses-test_task-fingerfootlips_bold_roi_mcf.nii.gz.par')
fig, axes = plt.subplots(2, 1, figsize=(15, 5))
axes[0].set_ylabel('rotation (radians)')
axes[0].plot(par[0:, :3])
axes[1].plot(par[0:, 3:])
axes[1].set_xlabel('time (TR)')
axes[1].set_ylabel('translation (mm)');
Explanation: But what did we do specifically? Well, let's investigate.
Motion Correction and Artifact Detection
How much did the subject move in the scanner and where there any outliers in the functional images?
End of explanation
# Showing the artifact detection output
from IPython.display import SVG
SVG(filename='/output/work_preproc/_subject_id_07/art/'
    'plot.asub-07_ses-test_task-fingerfootlips_bold_roi_mcf.svg')
Explanation: The motion parameters seems to look ok. What about the detection of artifacts?
End of explanation
outliers = np.loadtxt('/output/work_preproc/_subject_id_07/art/'
                      'art.asub-07_ses-test_task-fingerfootlips_bold_roi_mcf_outliers.txt')
list(outliers.astype('int'))
Explanation: Which volumes are problematic?
End of explanation
from nilearn import image as nli
from nilearn.plotting import plot_stat_map
%matplotlib inline
output = '/output/work_preproc/_subject_id_07/'
Explanation: Masks and Probability maps
Let's see what all the masks and probability maps look like. For this, we will use nilearn's plot_anat function.
End of explanation
anat = output + 'gunzip_anat/sub-07_ses-test_T1w.nii'
plot_stat_map(
    output + 'segment/c1sub-07_ses-test_T1w.nii', title='GM prob. map',  cmap=plt.cm.magma,
    threshold=0.5, bg_img=anat, display_mode='z', cut_coords=range(-35, 15, 10), dim=-1);
plot_stat_map(
    output + 'segment/c2sub-07_ses-test_T1w.nii', title='WM prob. map', cmap=plt.cm.magma,
    threshold=0.5, bg_img=anat, display_mode='z', cut_coords=range(-35, 15, 10), dim=-1);
plot_stat_map(
    output + 'segment/c3sub-07_ses-test_T1w.nii', title='CSF prob. map', cmap=plt.cm.magma,
    threshold=0.5, bg_img=anat, display_mode='z', cut_coords=range(-35, 15, 10), dim=-1);
Explanation: First, let's look at the tissue probability maps.
End of explanation
plot_stat_map(
    output + 'mask_GM/c1sub-07_ses-test_T1w_flirt_thresh.nii', title='dilated GM Mask', cmap=plt.cm.magma,
    threshold=0.5, bg_img=anat, display_mode='z', cut_coords=range(-35, 15, 10), dim=-1);
Explanation: And how does the gray matter mask look like that we used on the functional images?
End of explanation
from nilearn import image as nli
from nilearn.plotting import plot_epi
%matplotlib inline
output = '/output/work_preproc/_subject_id_07/'
plot_epi(output + 'mcflirt/asub-07_ses-test_task-fingerfootlips_bold_roi_mcf.nii.gz_mean_reg.nii.gz',
         title='Motion Corrected mean image', display_mode='z', cut_coords=range(-40, 21, 15),
         cmap=plt.cm.viridis);
mean = nli.mean_img(output + 'applywarp/asub-07_ses-test_task-fingerfootlips_bold_roi_mcf_flirt.nii')
plot_epi(mean, title='Coregistred mean image', display_mode='z', cut_coords=range(-40, 21, 15),
         cmap=plt.cm.viridis);
mean = nli.mean_img('/output/work_preproc/susan/_subject_id_07/smooth/mapflow/_smooth0/'
                    'asub-07_ses-test_task-fingerfootlips_bold_roi_mcf_flirt_smooth.nii.gz')
plot_epi(mean, title='Smoothed mean image', display_mode='z', cut_coords=range(-40, 21, 15),
         cmap=plt.cm.viridis);
mean = nli.mean_img(output + 'mask_func/mapflow/_mask_func0/'
                    'asub-07_ses-test_task-fingerfootlips_bold_roi_mcf_flirt_smooth_masked.nii')
plot_epi(mean, title='Masked mean image', display_mode='z', cut_coords=range(-40, 21, 15),
         cmap=plt.cm.viridis);
plot_epi(output + 'detrend/mean.nii.gz', title='Detrended mean image', display_mode='z',
         cut_coords=range(-40, 21, 15), cmap=plt.cm.viridis);
Explanation: Functional Image transformations
Let's also investigate the transformation that we applied to the functional images.
End of explanation
import nibabel as nb
%matplotlib inline
output = '/output/work_preproc/_subject_id_07/'
# Load the relevant datasets
mc = nb.load(output + 'applywarp/asub-07_ses-test_task-fingerfootlips_bold_roi_mcf_flirt.nii')
smooth = nb.load('/output/work_preproc/susan/_subject_id_07/smooth/mapflow/'
                 '_smooth0/asub-07_ses-test_task-fingerfootlips_bold_roi_mcf_flirt_smooth.nii.gz')
detrended_data = nb.load(output + 'detrend/detrend.nii.gz')
# Plot a representative voxel
x, y, z = 32, 34, 43
fig = plt.figure(figsize=(12, 4))
plt.plot(mc.get_data()[x, y, z, :])
plt.plot(smooth.get_data()[x, y, z, :])
plt.plot(detrended_data.get_data()[x, y, z, :])
plt.legend(['motion corrected', 'smoothed', 'detrended']);
Explanation: That's all nice and beautiful, but what did smoothing and detrending actually do to the data?
End of explanation
from nipype.interfaces.io import DataSink
# Initiate the datasink node
output_folder = 'datasink_handson'
datasink = Node(DataSink(base_directory='/output/',
                         container=output_folder),
                name="datasink")
Explanation: Data output with DataSink
The results look fine, but we don't need all those temporary files. So let's use Datasink to keep only those files that we actually need for the 1st and 2nd level analysis.
End of explanation
# Connect nodes to datasink here
preproc.connect([(art, datasink, [('outlier_files', 'preproc.@outlier_files'),
                                  ('plot_files', 'preproc.@plot_files')]),
                 (mcflirt, datasink, [('par_file', 'preproc.@par')]),
                 (detrend, datasink, [('detrended_file', 'preproc.@func')]),
                 ])
Explanation: Now the next step is to specify all the output that we want to keep in our output folder output. Make sure to keep:
- from the artifact detection node the outlier file as well as the outlier plot
- from the motion correction node the motion parameters
- from the last node, the detrended functional image
End of explanation
preproc.run('MultiProc', plugin_args={'n_procs': 4})
Explanation: Run the workflow
After adding the datasink folder, let's run the preprocessing workflow again.
End of explanation
!tree /output/datasink_handson -I '*js|*json|*pklz|_report|*dot|*html|*txt|*.m'
Explanation: Let's look now at the output of this datasink folder.
End of explanation
## Use the following substitutions for the DataSink output
substitutions = [('asub', 'sub'),
                 ('_ses-test_task-fingerfootlips_bold_roi_mcf', ''),
                 ('.nii.gz.par', '.par'),
                 ]
# To get rid of the folder '_subject_id_07' and renaming detrend
substitutions += [('_subject_id_%s/detrend' % s,
                   '_subject_id_%s/sub-%s_detrend' % (s, s)) for s in subject_list]
substitutions += [('_subject_id_%s/' % s, '') for s in subject_list]
datasink.inputs.substitutions = substitutions
Explanation: Much better! But we're still not there yet. There are many unnecessary file specifiers that we can get rid off. To do so, we can use DataSink's substitutions parameter. For this, we create a list of tuples: on the left, we specify the string that we want to replace and on the right, with what we want to replace it with.
End of explanation
# Delets the current output folder
!rm -rf /output/datasink_handson
# Runs the preprocessing workflow again, this time with substitutions
preproc.run('MultiProc', plugin_args={'n_procs': 4})
!tree /output/datasink_handson -I '*js|*json|*pklz|_report|*dot|*html|*.m'
Explanation: Before we run the preprocessing workflow again, let's first delete the current output folder:
End of explanation
# Update 'subject_list' and its dependencies here
subject_list = ['02', '03', '04', '07', '08', '09']
sf.iterables = [('subject_id', subject_list)]
# To get rid of the folder '_subject_id_02' and renaming detrend
substitutions += [('_subject_id_%s/detrend' % s,
                   '_subject_id_%s/sub-%s_detrend' % (s, s)) for s in subject_list]
substitutions += [('_subject_id_%s/' % s, '') for s in subject_list]
datasink.inputs.substitutions = substitutions
Explanation: Run Preprocessing workflow on 6 right-handed subjects
Perfect! Now let's run the whole workflow for right-handed subjects. For this, you just need to change the subject_list variable and run again the places where this variable is used (i.e. sf.iterables and in DataSink substitutions.
End of explanation
# Runs the preprocessing workflow again, this time with substitutions
preproc.run('MultiProc', plugin_args={'n_procs': 4})
Explanation: Now we can run the workflow again, this time for all right handed subjects in parallel.
End of explanation
<END_TASK>