{"action":"opened","issue":{"url":"https://api.github.com/repos/common-workflow-language/cwltool/issues/17","labels_url":"https://api.github.com/repos/common-workflow-language/cwltool/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/common-workflow-language/cwltool/issues/17/comments","events_url":"https://api.github.com/repos/common-workflow-language/cwltool/issues/17/events","html_url":"https://github.com/common-workflow-language/cwltool/issues/17","id":114928789,"number":17,"title":"CWLtool issue with passing a list of scatter/gathered outputs into a program that accepts a list","user":{"login":"jstjohn","id":459014,"avatar_url":"https://avatars.githubusercontent.com/u/459014?v=3","gravatar_id":"","url":"https://api.github.com/users/jstjohn","html_url":"https://github.com/jstjohn","followers_url":"https://api.github.com/users/jstjohn/followers","following_url":"https://api.github.com/users/jstjohn/following{/other_user}","gists_url":"https://api.github.com/users/jstjohn/gists{/gist_id}","starred_url":"https://api.github.com/users/jstjohn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jstjohn/subscriptions","organizations_url":"https://api.github.com/users/jstjohn/orgs","repos_url":"https://api.github.com/users/jstjohn/repos","events_url":"https://api.github.com/users/jstjohn/events{/privacy}","received_events_url":"https://api.github.com/users/jstjohn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-11-03T22:51:29Z","updated_at":"2015-11-03T22:51:29Z","closed_at":null,"body":"This works:\r\n\r\n```\r\n#!/usr/bin/env cwl-runner\r\nclass: Workflow\r\nrequirements:\r\n  - class: ScatterFeatureRequirement\r\ninputs:\r\n  - id: \"#mutation\"\r\n    type: string\r\n  - id: \"#normalin\"\r\n    type: File\r\n  - id: \"#tumorin\"\r\n    type: File\r\noutputs:\r\n  - id: \"#outfile\"\r\n    type: File\r\n    source: \"#wc.outfile\"\r\nsteps:\r\n  - id: \"#greptumor\"\r\n    run: {import: grep.cwl.yaml}\r\n    #scatter: \"#grep.infile\"\r\n    inputs:\r\n      - id: \"#grep.pattern\"\r\n        source: \"#mutation\"\r\n      - id: \"#grep.infile\"\r\n        source: \"#tumorin\"\r\n    outputs:\r\n      - id: \"#greptumor.outfile\"\r\n  - id: \"#grepnormal\"\r\n    run: {import: grep.cwl.yaml}\r\n    #scatter: \"#grep.infile\"\r\n    inputs:\r\n      - id: \"#grep.pattern\"\r\n        source: \"#mutation\"\r\n      - id: \"#grep.infile\"\r\n        source: \"#normalin\"\r\n    outputs:\r\n      - id: \"#grepnormal.outfile\"\r\n  - id: \"#wc\"\r\n    run: {import: wc.cwl.yaml}\r\n    inputs:\r\n      - id: \"#wc.infile\"\r\n        source: [\"#grepnormal.outfile\",  \"#greptumor.outfile\"]\r\n    outputs:\r\n      - id: \"#wc.outfile\"\r\n```\r\n\r\nThis does not work, when I try to have it work on multiple normal/tumor input files into grep scattered:\r\n\r\n```\r\n#!/usr/bin/env cwl-runner\r\nclass: Workflow\r\nrequirements:\r\n  - class: ScatterFeatureRequirement\r\ninputs:\r\n  - id: \"#mutation\"\r\n    type: string\r\n  - id: \"#normalin\"\r\n    type: {type: array, items: File}\r\n  - id: \"#tumorin\"\r\n    type: {type: array, items: File}\r\noutputs:\r\n  - id: \"#outfile\"\r\n    type: File\r\n    source: \"#wc.outfile\"\r\nsteps:\r\n  - id: \"#greptumor\"\r\n    run: {import: grep.cwl.yaml}\r\n    scatter: \"#grep.infile\"\r\n    inputs:\r\n      - id: \"#grep.pattern\"\r\n        source: \"#mutation\"\r\n      - id: \"#grep.infile\"\r\n        source: \"#tumorin\"\r\n    outputs:\r\n      - id: \"#greptumor.outfile\"\r\n  - id: \"#grepnormal\"\r\n    run: {import: grep.cwl.yaml}\r\n    scatter: \"#grep.infile\"\r\n    inputs:\r\n      - id: \"#grep.pattern\"\r\n        source: \"#mutation\"\r\n      - id: \"#grep.infile\"\r\n        source: \"#normalin\"\r\n    outputs:\r\n      - id: \"#grepnormal.outfile\"\r\n  - id: \"#wc\"\r\n    run: {import: wc.cwl.yaml}\r\n    inputs:\r\n      - id: \"#wc.infile\"\r\n        source: [\"#grepnormal.outfile\",  \"#greptumor.outfile\"]\r\n    outputs:\r\n      - id: \"#wc.outfile\"\r\n```\r\n\r\n\r\nInterestingly though, this small change does work, when you only pass one of the scatter/gathered inputs into the final wc.\r\n\r\n```\r\n#!/usr/bin/env cwl-runner\r\nclass: Workflow\r\nrequirements:\r\n  - class: ScatterFeatureRequirement\r\ninputs:\r\n  - id: \"#mutation\"\r\n    type: string\r\n  - id: \"#normalin\"\r\n    type: {type: array, items: File}\r\n  - id: \"#tumorin\"\r\n    type: {type: array, items: File}\r\noutputs:\r\n  - id: \"#outfile\"\r\n    type: File\r\n    source: \"#wc.outfile\"\r\nsteps:\r\n  - id: \"#greptumor\"\r\n    run: {import: grep.cwl.yaml}\r\n    scatter: \"#grep.infile\"\r\n    inputs:\r\n      - id: \"#grep.pattern\"\r\n        source: \"#mutation\"\r\n      - id: \"#grep.infile\"\r\n        source: \"#tumorin\"\r\n    outputs:\r\n      - id: \"#greptumor.outfile\"\r\n  - id: \"#grepnormal\"\r\n    run: {import: grep.cwl.yaml}\r\n    scatter: \"#grep.infile\"\r\n    inputs:\r\n      - id: \"#grep.pattern\"\r\n        source: \"#mutation\"\r\n      - id: \"#grep.infile\"\r\n        source: \"#normalin\"\r\n    outputs:\r\n      - id: \"#grepnormal.outfile\"\r\n  - id: \"#wc\"\r\n    run: {import: wc.cwl.yaml}\r\n    inputs:\r\n      - id: \"#wc.infile\"\r\n        source: [\"#grepnormal.outfile\"]\r\n    outputs:\r\n      - id: \"#wc.outfile\"\r\n```\r\n\r\nSo it seems to me that this is an issue only when you use the program accepting an array of inputs and you want to pass two of those scatter/gathered things into a final output. Maybe the data structure being output in the scatter/gather is weird when you put it into a list? The error is this about a dict being expected:\r\n\r\n```\r\n> cwltool mutationfinder3.cwl.yaml --mutation GCATCCA --normalin normal.fastq --tumorin tumor.fastq \r\n/anaconda/bin/cwltool 1.0.20151026181844\r\n[job 4386765520] /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpm8fJ5E$ grep GCATCCA /Users/john/workspaces/commonwl_examples/t790m_detector/normal.fastq > /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpm8fJ5E/out.txt\r\n[job 4386765840] /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpUclOsb$ grep GCATCCA /Users/john/workspaces/commonwl_examples/t790m_detector/tumor.fastq > /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpUclOsb/out.txt\r\nWorkflow error:\r\n  Error validating input record, could not validate field `infile` because\r\n  At position 0\r\n    `[{'checksum': 'sha1$e62341a47da9cb155ec0f531b525a97d2bd86455',\r\n      'class': 'File',\r\n      'path': '/var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpm8fJ5E/out.txt'[...]`\r\n     is not a dict\r\n```\r\n"}}