{"action":"created","comment":{"url":"https://api.github.com/repos/biocore/qiita/pulls/comments/41657063","id":41657063,"diff_hunk":"@@ -872,50 +853,68 @@ def generate_demultiplexed_fastq(self, rewrite_fastq=False, mtime=None):\n \n         Notes\n         -----\n-        - As a performace feature, this method will check if self.ebi_dir\n+        - As a performace feature, this method will check if self.full_ebi_dir\n         already exists and, if it does, the script will assume that in a\n         previous execution this step was performed correctly and will simply\n-        read the file names from self.ebi_dir\n+        read the file names from self.full_ebi_dir\n         - When the object is created (init), samples, samples_prep and\n         sample_demux_fps hold values for all available samples in the database.\n         Here some of those values will be deleted (del's, within the loops) for\n         those cases where the fastq.gz files weren't written or exist. This is\n         an indication that they had no sequences and this kind of files are not\n         accepted in EBI\n+\n+        Raises\n+        ------\n+        EBISubmissionError\n+            - The demux file couldn't be read\n+            - All samples are removed\n         \"\"\"\n         ppd = self.preprocessed_data\n \n-        if not isdir(self.ebi_dir) or rewrite_fastq:\n-            makedirs(self.ebi_dir)\n+        dir_not_exists = not isdir(self.full_ebi_dir)\n+        if dir_not_exists or rewrite_fastq:\n+            if dir_not_exists:\n+                makedirs(self.full_ebi_dir)\n+                filetype_id = convert_to_id('directory', 'filepath_type')\n+                insert_filepaths([(self.full_ebi_dir, filetype_id)], -1,\n+                                 'preprocessed_data', 'filepath',\n+                                 move_files=False)\n \n             demux = [path for _, path, ftype in ppd.get_filepaths()\n                      if ftype == 'preprocessed_demux'][0]\n \n             demux_samples = set()\n-            with open_file(demux) as demux_fh:\n-                for s, i in to_per_sample_ascii(demux_fh,\n-                                                self.prep_template.keys()):\n-                    sample_fp = self.sample_demux_fps[s]\n-                    wrote_sequences = False\n-                    with GzipFile(sample_fp, mode='w', mtime=mtime) as fh:\n-                        for record in i:\n-                            fh.write(record)\n-                            wrote_sequences = True\n-\n-                    if wrote_sequences:\n-                        demux_samples.add(s)\n-                    else:\n-                        del(self.samples[s])\n-                        del(self.samples_prep[s])\n-                        del(self.sample_demux_fps[s])\n-                        remove(sample_fp)\n-\n+            try:","path":"qiita_ware/ebi.py","position":210,"original_position":210,"commit_id":"6417aa6ad74b9bc109dce473722d271a9c32b115","original_commit_id":"6417aa6ad74b9bc109dce473722d271a9c32b115","user":{"login":"antgonza","id":2014559,"avatar_url":"https://avatars.githubusercontent.com/u/2014559?v=3","gravatar_id":"","url":"https://api.github.com/users/antgonza","html_url":"https://github.com/antgonza","followers_url":"https://api.github.com/users/antgonza/followers","following_url":"https://api.github.com/users/antgonza/following{/other_user}","gists_url":"https://api.github.com/users/antgonza/gists{/gist_id}","starred_url":"https://api.github.com/users/antgonza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antgonza/subscriptions","organizations_url":"https://api.github.com/users/antgonza/orgs","repos_url":"https://api.github.com/users/antgonza/repos","events_url":"https://api.github.com/users/antgonza/events{/privacy}","received_events_url":"https://api.github.com/users/antgonza/received_events","type":"User","site_admin":false},"body":"Good question. While writing tests I realized that one issue is that one posible issue is that the demux file doesn't exist or there are errors while reading it. Thus, that try covers any of those possible issues. I tried to isolate them but was really hard as the return is an iterator and there is no easy way to know if empty or not.","created_at":"2015-10-09T17:38:54Z","updated_at":"2015-10-09T17:38:54Z","html_url":"https://github.com/biocore/qiita/pull/1488#discussion_r41657063","pull_request_url":"https://api.github.com/repos/biocore/qiita/pulls/1488","_links":{"self":{"href":"https://api.github.com/repos/biocore/qiita/pulls/comments/41657063"},"html":{"href":"https://github.com/biocore/qiita/pull/1488#discussion_r41657063"},"pull_request":{"href":"https://api.github.com/repos/biocore/qiita/pulls/1488"}}},"pull_request":{"url":"https://api.github.com/repos/biocore/qiita/pulls/1488","id":46947808,"html_url":"https://github.com/biocore/qiita/pull/1488","diff_url":"https://github.com/biocore/qiita/pull/1488.diff","patch_url":"https://github.com/biocore/qiita/pull/1488.patch","issue_url":"https://api.github.com/repos/biocore/qiita/issues/1488","number":1488,"state":"open","locked":false,"title":"WIP: Next steps","user":{"login":"antgonza","id":2014559,"avatar_url":"https://avatars.githubusercontent.com/u/2014559?v=3","gravatar_id":"","url":"https://api.github.com/users/antgonza","html_url":"https://github.com/antgonza","followers_url":"https://api.github.com/users/antgonza/followers","following_url":"https://api.github.com/users/antgonza/following{/other_user}","gists_url":"https://api.github.com/users/antgonza/gists{/gist_id}","starred_url":"https://api.github.com/users/antgonza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antgonza/subscriptions","organizations_url":"https://api.github.com/users/antgonza/orgs","repos_url":"https://api.github.com/users/antgonza/repos","events_url":"https://api.github.com/users/antgonza/events{/privacy}","received_events_url":"https://api.github.com/users/antgonza/received_events","type":"User","site_admin":false},"body":"This is based on https://github.com/biocore/qiita/pull/1481 so that should be merged first.\r\n\r\nHere I'm fixing the last standing points in the joy list, so far:\r\n- Fix #1479\r\n- In the template documentation changing that sample description is a required field for Qiita deploy to a requirement for EBI, as this is required for submissions.\r\n- Removing 2 cfg variables that are not needed anymore with the new EBI system. Today we got a notification that the old system will be out in 2 months.\r\n- Changes so tests pass in the main branch.","created_at":"2015-10-06T19:39:53Z","updated_at":"2015-10-09T17:38:54Z","closed_at":null,"merged_at":null,"merge_commit_sha":"551fbb23465d88adc4b75dbdee439a074d50ab37","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/biocore/qiita/pulls/1488/commits","review_comments_url":"https://api.github.com/repos/biocore/qiita/pulls/1488/comments","review_comment_url":"https://api.github.com/repos/biocore/qiita/pulls/comments{/number}","comments_url":"https://api.github.com/repos/biocore/qiita/issues/1488/comments","statuses_url":"https://api.github.com/repos/biocore/qiita/statuses/6417aa6ad74b9bc109dce473722d271a9c32b115","head":{"label":"antgonza:next-joy-list","ref":"next-joy-list","sha":"6417aa6ad74b9bc109dce473722d271a9c32b115","user":{"login":"antgonza","id":2014559,"avatar_url":"https://avatars.githubusercontent.com/u/2014559?v=3","gravatar_id":"","url":"https://api.github.com/users/antgonza","html_url":"https://github.com/antgonza","followers_url":"https://api.github.com/users/antgonza/followers","following_url":"https://api.github.com/users/antgonza/following{/other_user}","gists_url":"https://api.github.com/users/antgonza/gists{/gist_id}","starred_url":"https://api.github.com/users/antgonza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antgonza/subscriptions","organizations_url":"https://api.github.com/users/antgonza/orgs","repos_url":"https://api.github.com/users/antgonza/repos","events_url":"https://api.github.com/users/antgonza/events{/privacy}","received_events_url":"https://api.github.com/users/antgonza/received_events","type":"User","site_admin":false},"repo":{"id":16964638,"name":"qiita","full_name":"antgonza/qiita","owner":{"login":"antgonza","id":2014559,"avatar_url":"https://avatars.githubusercontent.com/u/2014559?v=3","gravatar_id":"","url":"https://api.github.com/users/antgonza","html_url":"https://github.com/antgonza","followers_url":"https://api.github.com/users/antgonza/followers","following_url":"https://api.github.com/users/antgonza/following{/other_user}","gists_url":"https://api.github.com/users/antgonza/gists{/gist_id}","starred_url":"https://api.github.com/users/antgonza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antgonza/subscriptions","organizations_url":"https://api.github.com/users/antgonza/orgs","repos_url":"https://api.github.com/users/antgonza/repos","events_url":"https://api.github.com/users/antgonza/events{/privacy}","received_events_url":"https://api.github.com/users/antgonza/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/antgonza/qiita","description":"Qiita - A QIIME databasing effort","fork":true,"url":"https://api.github.com/repos/antgonza/qiita","forks_url":"https://api.github.com/repos/antgonza/qiita/forks","keys_url":"https://api.github.com/repos/antgonza/qiita/keys{/key_id}","collaborators_url":"https://api.github.com/repos/antgonza/qiita/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/antgonza/qiita/teams","hooks_url":"https://api.github.com/repos/antgonza/qiita/hooks","issue_events_url":"https://api.github.com/repos/antgonza/qiita/issues/events{/number}","events_url":"https://api.github.com/repos/antgonza/qiita/events","assignees_url":"https://api.github.com/repos/antgonza/qiita/assignees{/user}","branches_url":"https://api.github.com/repos/antgonza/qiita/branches{/branch}","tags_url":"https://api.github.com/repos/antgonza/qiita/tags","blobs_url":"https://api.github.com/repos/antgonza/qiita/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/antgonza/qiita/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/antgonza/qiita/git/refs{/sha}","trees_url":"https://api.github.com/repos/antgonza/qiita/git/trees{/sha}","statuses_url":"https://api.github.com/repos/antgonza/qiita/statuses/{sha}","languages_url":"https://api.github.com/repos/antgonza/qiita/languages","stargazers_url":"https://api.github.com/repos/antgonza/qiita/stargazers","contributors_url":"https://api.github.com/repos/antgonza/qiita/contributors","subscribers_url":"https://api.github.com/repos/antgonza/qiita/subscribers","subscription_url":"https://api.github.com/repos/antgonza/qiita/subscription","commits_url":"https://api.github.com/repos/antgonza/qiita/commits{/sha}","git_commits_url":"https://api.github.com/repos/antgonza/qiita/git/commits{/sha}","comments_url":"https://api.github.com/repos/antgonza/qiita/comments{/number}","issue_comment_url":"https://api.github.com/repos/antgonza/qiita/issues/comments{/number}","contents_url":"https://api.github.com/repos/antgonza/qiita/contents/{+path}","compare_url":"https://api.github.com/repos/antgonza/qiita/compare/{base}...{head}","merges_url":"https://api.github.com/repos/antgonza/qiita/merges","archive_url":"https://api.github.com/repos/antgonza/qiita/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/antgonza/qiita/downloads","issues_url":"https://api.github.com/repos/antgonza/qiita/issues{/number}","pulls_url":"https://api.github.com/repos/antgonza/qiita/pulls{/number}","milestones_url":"https://api.github.com/repos/antgonza/qiita/milestones{/number}","notifications_url":"https://api.github.com/repos/antgonza/qiita/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/antgonza/qiita/labels{/name}","releases_url":"https://api.github.com/repos/antgonza/qiita/releases{/id}","created_at":"2014-02-18T21:38:48Z","updated_at":"2015-05-04T02:58:47Z","pushed_at":"2015-10-09T14:27:33Z","git_url":"git://github.com/antgonza/qiita.git","ssh_url":"git@github.com:antgonza/qiita.git","clone_url":"https://github.com/antgonza/qiita.git","svn_url":"https://github.com/antgonza/qiita","homepage":"","size":128132,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"biocore:ebi-improvements","ref":"ebi-improvements","sha":"e26156433b6ac6139302d52ac68e50cb395d65e9","user":{"login":"biocore","id":6241857,"avatar_url":"https://avatars.githubusercontent.com/u/6241857?v=3","gravatar_id":"","url":"https://api.github.com/users/biocore","html_url":"https://github.com/biocore","followers_url":"https://api.github.com/users/biocore/followers","following_url":"https://api.github.com/users/biocore/following{/other_user}","gists_url":"https://api.github.com/users/biocore/gists{/gist_id}","starred_url":"https://api.github.com/users/biocore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/biocore/subscriptions","organizations_url":"https://api.github.com/users/biocore/orgs","repos_url":"https://api.github.com/users/biocore/repos","events_url":"https://api.github.com/users/biocore/events{/privacy}","received_events_url":"https://api.github.com/users/biocore/received_events","type":"Organization","site_admin":false},"repo":{"id":13131245,"name":"qiita","full_name":"biocore/qiita","owner":{"login":"biocore","id":6241857,"avatar_url":"https://avatars.githubusercontent.com/u/6241857?v=3","gravatar_id":"","url":"https://api.github.com/users/biocore","html_url":"https://github.com/biocore","followers_url":"https://api.github.com/users/biocore/followers","following_url":"https://api.github.com/users/biocore/following{/other_user}","gists_url":"https://api.github.com/users/biocore/gists{/gist_id}","starred_url":"https://api.github.com/users/biocore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/biocore/subscriptions","organizations_url":"https://api.github.com/users/biocore/orgs","repos_url":"https://api.github.com/users/biocore/repos","events_url":"https://api.github.com/users/biocore/events{/privacy}","received_events_url":"https://api.github.com/users/biocore/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/biocore/qiita","description":"Qiita - A QIIME databasing effort","fork":false,"url":"https://api.github.com/repos/biocore/qiita","forks_url":"https://api.github.com/repos/biocore/qiita/forks","keys_url":"https://api.github.com/repos/biocore/qiita/keys{/key_id}","collaborators_url":"https://api.github.com/repos/biocore/qiita/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/biocore/qiita/teams","hooks_url":"https://api.github.com/repos/biocore/qiita/hooks","issue_events_url":"https://api.github.com/repos/biocore/qiita/issues/events{/number}","events_url":"https://api.github.com/repos/biocore/qiita/events","assignees_url":"https://api.github.com/repos/biocore/qiita/assignees{/user}","branches_url":"https://api.github.com/repos/biocore/qiita/branches{/branch}","tags_url":"https://api.github.com/repos/biocore/qiita/tags","blobs_url":"https://api.github.com/repos/biocore/qiita/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/biocore/qiita/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/biocore/qiita/git/refs{/sha}","trees_url":"https://api.github.com/repos/biocore/qiita/git/trees{/sha}","statuses_url":"https://api.github.com/repos/biocore/qiita/statuses/{sha}","languages_url":"https://api.github.com/repos/biocore/qiita/languages","stargazers_url":"https://api.github.com/repos/biocore/qiita/stargazers","contributors_url":"https://api.github.com/repos/biocore/qiita/contributors","subscribers_url":"https://api.github.com/repos/biocore/qiita/subscribers","subscription_url":"https://api.github.com/repos/biocore/qiita/subscription","commits_url":"https://api.github.com/repos/biocore/qiita/commits{/sha}","git_commits_url":"https://api.github.com/repos/biocore/qiita/git/commits{/sha}","comments_url":"https://api.github.com/repos/biocore/qiita/comments{/number}","issue_comment_url":"https://api.github.com/repos/biocore/qiita/issues/comments{/number}","contents_url":"https://api.github.com/repos/biocore/qiita/contents/{+path}","compare_url":"https://api.github.com/repos/biocore/qiita/compare/{base}...{head}","merges_url":"https://api.github.com/repos/biocore/qiita/merges","archive_url":"https://api.github.com/repos/biocore/qiita/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/biocore/qiita/downloads","issues_url":"https://api.github.com/repos/biocore/qiita/issues{/number}","pulls_url":"https://api.github.com/repos/biocore/qiita/pulls{/number}","milestones_url":"https://api.github.com/repos/biocore/qiita/milestones{/number}","notifications_url":"https://api.github.com/repos/biocore/qiita/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/biocore/qiita/labels{/name}","releases_url":"https://api.github.com/repos/biocore/qiita/releases{/id}","created_at":"2013-09-26T18:50:43Z","updated_at":"2015-09-20T19:09:27Z","pushed_at":"2015-10-09T14:14:51Z","git_url":"git://github.com/biocore/qiita.git","ssh_url":"git@github.com:biocore/qiita.git","clone_url":"https://github.com/biocore/qiita.git","svn_url":"https://github.com/biocore/qiita","homepage":"http://qiita.microbio.me","size":132756,"stargazers_count":24,"watchers_count":24,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":24,"mirror_url":null,"open_issues_count":318,"forks":24,"open_issues":318,"watchers":24,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/biocore/qiita/pulls/1488"},"html":{"href":"https://github.com/biocore/qiita/pull/1488"},"issue":{"href":"https://api.github.com/repos/biocore/qiita/issues/1488"},"comments":{"href":"https://api.github.com/repos/biocore/qiita/issues/1488/comments"},"review_comments":{"href":"https://api.github.com/repos/biocore/qiita/pulls/1488/comments"},"review_comment":{"href":"https://api.github.com/repos/biocore/qiita/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/biocore/qiita/pulls/1488/commits"},"statuses":{"href":"https://api.github.com/repos/biocore/qiita/statuses/6417aa6ad74b9bc109dce473722d271a9c32b115"}}}}