{"action":"created","issue":{"url":"https://api.github.com/repos/biocore/qiime/issues/1963","labels_url":"https://api.github.com/repos/biocore/qiime/issues/1963/labels{/name}","comments_url":"https://api.github.com/repos/biocore/qiime/issues/1963/comments","events_url":"https://api.github.com/repos/biocore/qiime/issues/1963/events","html_url":"https://github.com/biocore/qiime/issues/1963","id":64141954,"number":1963,"title":"filter_taxa_from_otu_table.py uninformative error","user":{"login":"wdwvt1","id":1048569,"avatar_url":"https://avatars.githubusercontent.com/u/1048569?v=3","gravatar_id":"","url":"https://api.github.com/users/wdwvt1","html_url":"https://github.com/wdwvt1","followers_url":"https://api.github.com/users/wdwvt1/followers","following_url":"https://api.github.com/users/wdwvt1/following{/other_user}","gists_url":"https://api.github.com/users/wdwvt1/gists{/gist_id}","starred_url":"https://api.github.com/users/wdwvt1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wdwvt1/subscriptions","organizations_url":"https://api.github.com/users/wdwvt1/orgs","repos_url":"https://api.github.com/users/wdwvt1/repos","events_url":"https://api.github.com/users/wdwvt1/events{/privacy}","received_events_url":"https://api.github.com/users/wdwvt1/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/biocore/qiime/labels/bug","name":"bug","color":"e10c02"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-03-25T01:19:50Z","updated_at":"2015-03-25T16:09:35Z","closed_at":null,"body":"When attempting to filter an OTU table down to only a specific taxonomy string (`-p` option), if the taxonomy is not found the script will error with a non-informative message. It will also create a malformed HDF5 table in the specified output location. \r\n\r\nHere is an example that creates a small table that produces the error:\r\n```python\r\nfrom numpy import array\r\nfrom biom.table import Table\r\nfrom h5py import File\r\ndata = array([[1,2,3],[4,5,6]])\r\noids = ['otu1', 'otu2']\r\nsids = ['s1', 's2', 's3']\r\nmd = [{'taxonomy': ['k__bacteria','p__firmicutes']}, {'taxonomy':['k__eukaryotes', 'p__something']}]\r\nbt = Table(data, oids, sids, observation_metadata=md)\r\n\r\no = File('/Users/wdwvt/Desktop/ex.biom', 'w')\r\nbt.to_hdf5(o, 'example')\r\no.close()\r\n```\r\n\r\nThen, attempt to filter with\r\n```bash\r\nfilter_taxa_from_otu_table.py -i /Users/wdwvt/Desktop/ex.biom -p 'something_not_in_there' -o /Users/wdwvt/Desktop/ex.f.biom\r\n```\r\nErrors with:\r\n```bash\r\nTraceback (most recent call last):\r\n  File \"/Users/wdwvt/src/qiime/scripts/filter_taxa_from_otu_table.py\", line 85, in <module>\r\n    main()\r\n  File \"/Users/wdwvt/src/qiime/scripts/filter_taxa_from_otu_table.py\", line 81, in main\r\n    write_biom_table(input_table, output_table_fp)\r\n  File \"/Users/wdwvt/src/qiime/qiime/util.py\", line 555, in write_biom_table\r\n    biom_table.to_hdf5(biom_file, generated_by, compress)\r\n  File \"/Users/wdwvt/src/biom-format/build/lib.macosx-10.5-x86_64-2.7/biom/table.py\", line 3481, in to_hdf5\r\n    self.group_metadata(axis='observation'), 'csr', compression)\r\n  File \"/Users/wdwvt/src/biom-format/build/lib.macosx-10.5-x86_64-2.7/biom/table.py\", line 3450, in axis_dump\r\n    for category in md[0]:\r\nIndexError: tuple index out of range\r\n```"},"comment":{"url":"https://api.github.com/repos/biocore/qiime/issues/comments/86098028","html_url":"https://github.com/biocore/qiime/issues/1963#issuecomment-86098028","issue_url":"https://api.github.com/repos/biocore/qiime/issues/1963","id":86098028,"user":{"login":"ElDeveloper","id":375307,"avatar_url":"https://avatars.githubusercontent.com/u/375307?v=3","gravatar_id":"","url":"https://api.github.com/users/ElDeveloper","html_url":"https://github.com/ElDeveloper","followers_url":"https://api.github.com/users/ElDeveloper/followers","following_url":"https://api.github.com/users/ElDeveloper/following{/other_user}","gists_url":"https://api.github.com/users/ElDeveloper/gists{/gist_id}","starred_url":"https://api.github.com/users/ElDeveloper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ElDeveloper/subscriptions","organizations_url":"https://api.github.com/users/ElDeveloper/orgs","repos_url":"https://api.github.com/users/ElDeveloper/repos","events_url":"https://api.github.com/users/ElDeveloper/events{/privacy}","received_events_url":"https://api.github.com/users/ElDeveloper/received_events","type":"User","site_admin":false},"created_at":"2015-03-25T16:09:35Z","updated_at":"2015-03-25T16:09:35Z","body":"If others agree to this solution, I can try putting it in place, but\nsince it affects the behavior of the whole package, I would rather be\ncautious about it.\n\nOn (Mar-25-15| 8:51), Daniel McDonald wrote:\n> I think so\n> \n> On Wed, Mar 25, 2015 at 9:28 AM, Yoshiki Vázquez Baeza <\n> notifications@github.com> wrote:\n> \n> > Yeah, so probably this should be done at the module level?\n> >\n> >\n> > On (Mar-25-15| 7:38), Daniel McDonald wrote:\n> > > One fix is for qiime to use bioms seterr and set empty='raise'\n> > > On Mar 25, 2015 00:39, \"Yoshiki Vázquez Baeza\" <notifications@github.com\n> > >\n> > > wrote:\n> > >\n> > > > @wdwvt1, I was able to reproduce the bug, and I think I found the\n> > > > culprit. See the links below. (1) biom's err.py module tags the check\n> > > > for \"empty tables\" as errors that can be \"ignored\" and (2) even if that\n> > > > was not the case the raised error is pretty bad from the context of a\n> > > > QIIME script.\n> > > >\n> > > > ```\n> > > > File\n> > > >\n> > \"/Users/yoshikivazquezbaeza/.virtualenvs/qiime-dev/bin/filter_taxa_from_otu_table.py\",\n> > > > line 10, in <module>\n> > > > execfile(__file__)\n> > > > File\n> > > >\n> > \"/Users/yoshikivazquezbaeza/git_sw/qiime/scripts/filter_taxa_from_otu_table.py\",\n> > > > line 87, in <module>\n> > > > main()\n> > > > File\n> > > >\n> > \"/Users/yoshikivazquezbaeza/git_sw/qiime/scripts/filter_taxa_from_otu_table.py\",\n> > > > line 79, in main\n> > > > input_table.filter(filter_fn, axis='observation')\n> > > > File\n> > > >\n> > \"/Users/yoshikivazquezbaeza/.virtualenvs/qiime-dev/lib/python2.7/site-packages/biom/table.py\",\n> > > > line 1869, in filter\n> > > > errcheck(table, 'empty')\n> > > > File\n> > > >\n> > \"/Users/yoshikivazquezbaeza/.virtualenvs/qiime-dev/lib/python2.7/site-packages/biom/err.py\",\n> > > > line 473, in errcheck\n> > > > raise ret\n> > > > biom.exception.TableException: Empty table!\n> > > > ```\n> > > >\n> > > > https://github.com/biocore/biom-format/blob/master/biom/err.py#L322\n> > > > https://github.com/biocore/biom-format/blob/master/biom/table.py#L1872\n> > > >\n> > > > I've opened an issue in biom to see if this was an intended behavior\n> > > > (and applications like QIIME have to change this at the global state)\n> > or\n> > > > a bug.\n> > > >\n> > > > https://github.com/biocore/biom-format/issues/619\n> > > >\n> > > > On (Mar-24-15|18:19), Will Van Treuren wrote:\n> > > > > When attempting to filter an OTU table down to only a specific\n> > taxonomy\n> > > > string (`-p` option), if the taxonomy is not found the script will\n> > error\n> > > > with a non-informative message. It will also create a malformed HDF5\n> > table\n> > > > in the specified output location.\n> > > > >\n> > > > > Here is an example that creates a small table that produces the\n> > error:\n> > > > > ```python\n> > > > > from numpy import array\n> > > > > from biom.table import Table\n> > > > > from h5py import File\n> > > > > data = array([[1,2,3],[4,5,6]])\n> > > > > oids = ['otu1', 'otu2']\n> > > > > sids = ['s1', 's2', 's3']\n> > > > > md = [{'taxonomy': ['k__bacteria','p__firmicutes']},\n> > > > {'taxonomy':['k__eukaryotes', 'p__something']}]\n> > > > > bt = Table(data, oids, sids, observation_metadata=md)\n> > > > >\n> > > > > o = File('/Users/wdwvt/Desktop/ex.biom', 'w')\n> > > > > bt.to_hdf5(o, 'example')\n> > > > > o.close()\n> > > > > ```\n> > > > >\n> > > > > Then, attempt to filter with\n> > > > > ```bash\n> > > > > filter_taxa_from_otu_table.py -i /Users/wdwvt/Desktop/ex.biom -p\n> > > > 'something_not_in_there' -o /Users/wdwvt/Desktop/ex.f.biom\n> > > > > ```\n> > > > > Errors with:\n> > > > > ```bash\n> > > > > Traceback (most recent call last):\n> > > > > File \"/Users/wdwvt/src/qiime/scripts/filter_taxa_from_otu_table.py\",\n> > > > line 85, in <module>\n> > > > > main()\n> > > > > File \"/Users/wdwvt/src/qiime/scripts/filter_taxa_from_otu_table.py\",\n> > > > line 81, in main\n> > > > > write_biom_table(input_table, output_table_fp)\n> > > > > File \"/Users/wdwvt/src/qiime/qiime/util.py\", line 555, in\n> > > > write_biom_table\n> > > > > biom_table.to_hdf5(biom_file, generated_by, compress)\n> > > > > File\n> > > >\n> > \"/Users/wdwvt/src/biom-format/build/lib.macosx-10.5-x86_64-2.7/biom/table.py\",\n> > > > line 3481, in to_hdf5\n> > > > > self.group_metadata(axis='observation'), 'csr', compression)\n> > > > > File\n> > > >\n> > \"/Users/wdwvt/src/biom-format/build/lib.macosx-10.5-x86_64-2.7/biom/table.py\",\n> > > > line 3450, in axis_dump\n> > > > > for category in md[0]:\n> > > > > IndexError: tuple index out of range\n> > > > > ```\n> > > > >\n> > > > > ---\n> > > > > Reply to this email directly or view it on GitHub:\n> > > > > https://github.com/biocore/qiime/issues/1963\n> > > >\n> > > > —\n> > > > Reply to this email directly or view it on GitHub\n> > > > <https://github.com/biocore/qiime/issues/1963#issuecomment-85876663>.\n> > > >\n> > >\n> > >\n> > > ---\n> > > Reply to this email directly or view it on GitHub:\n> > > https://github.com/biocore/qiime/issues/1963#issuecomment-86055390\n> >\n> > —\n> > Reply to this email directly or view it on GitHub\n> > <https://github.com/biocore/qiime/issues/1963#issuecomment-86082068>.\n> >\n> \n> \n> ---\n> Reply to this email directly or view it on GitHub:\n> https://github.com/biocore/qiime/issues/1963#issuecomment-86092250\n"}}