{"action":"created","issue":{"url":"https://api.github.com/repos/biocore/biom-format/issues/688","labels_url":"https://api.github.com/repos/biocore/biom-format/issues/688/labels{/name}","comments_url":"https://api.github.com/repos/biocore/biom-format/issues/688/comments","events_url":"https://api.github.com/repos/biocore/biom-format/issues/688/events","html_url":"https://github.com/biocore/biom-format/issues/688","id":124563878,"number":688,"title":"biom.table.Table.transform unexpected behaviour","user":{"login":"grayfall","id":6906110,"avatar_url":"https://avatars.githubusercontent.com/u/6906110?v=3","gravatar_id":"","url":"https://api.github.com/users/grayfall","html_url":"https://github.com/grayfall","followers_url":"https://api.github.com/users/grayfall/followers","following_url":"https://api.github.com/users/grayfall/following{/other_user}","gists_url":"https://api.github.com/users/grayfall/gists{/gist_id}","starred_url":"https://api.github.com/users/grayfall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grayfall/subscriptions","organizations_url":"https://api.github.com/users/grayfall/orgs","repos_url":"https://api.github.com/users/grayfall/repos","events_url":"https://api.github.com/users/grayfall/events{/privacy}","received_events_url":"https://api.github.com/users/grayfall/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-01-01T22:24:03Z","updated_at":"2016-01-01T23:03:04Z","closed_at":null,"body":"I'm working on a function to merge samples in BIOM tables. The code is heavily documented. The problem is described below the function's code. \r\n\r\n    def merge_samples(biom_table, sample_mapping):\r\n        # TODO finalize docs. Stress that unlisted samples will be thrown away\r\n        \"\"\"\r\n\r\n        :type biom_table: biom.table.Table\r\n        :param sample_mapping: a dict that maps sample IDs to lists of IDs in\r\n         biom_table to be merged (e.g. replicates), e.g. {merged_sample_id: [a set\r\n         of old samples to merge]}\r\n        :type sample_mapping: dict[str, collections.Sequence[str]]\r\n        :return:\r\n        \"\"\"\r\n        def pick_temp_sample_id_and_invert(_sample_mapping):\r\n            \"\"\"\r\n            For each supersample pick the first listed subsample to use as a\r\n            temporary ID and invert the mapping\r\n            :type _sample_mapping: dict[str, collections.Sequence[str]]\r\n            :param _sample_mapping: {merged_id: [a set of old samples to merge]}\r\n            :rtype: dict[str, str]\r\n            \"\"\"\r\n            return {subsamples[0]: supersample for supersample, subsamples\r\n                    in _sample_mapping.iteritems()}\r\n\r\n        def reduce_subsamples(_sample_mapping):\r\n            \"\"\"\r\n            Sum all subsamples of a supersample\r\n            :type _sample_mapping: dict[str, collections.Sequence[str]]\r\n            :rtype: dict[str, np.array]\r\n            \"\"\"\r\n            return {\r\n                supersample: reduce(op.add,\r\n                                    (biom_table.data(_id) for _id in subsamples))\r\n                for supersample, subsamples in _sample_mapping.iteritems()\r\n            }\r\n\r\n        # Merging subsamples. Picking temporary sample IDs and inverting the mapping\r\n        # (required by `transform_func`) and dropping excess samples (i.e. that\r\n        #  were merged)\r\n        reduced_subsamples = reduce_subsamples(sample_mapping)\r\n        temp_ids_to_new_ids = pick_temp_sample_id_and_invert(sample_mapping)\r\n        ids_to_keep = temp_ids_to_new_ids.keys()\r\n        filtered_table = biom_table.filter(ids_to_keep, inplace=False)\r\n\r\n        def transform_func(data, _id, metadata):\r\n            \"\"\"\r\n            Returns a reduced set of subsamples for each temporary _id\r\n            :rtype: np.array\r\n            \"\"\"\r\n            return reduced_subsamples[temp_ids_to_new_ids[_id]]\r\n\r\n        # Transforming biom_table, i.e. replacing data of temp_ids with summed\r\n        # arrays\r\n        tranformed_biom = filtered_table.transform(transform_func, axis=\"sample\")\r\n\r\n        # Renaming merged samples \r\n        return tranformed_biom.update_ids(temp_ids_to_new_ids)\r\n\r\nHere are the data I use for unit testing\r\n\r\n    test_data = np.arange(4).reshape((1, 4))\r\n    sample_ids = [\"A1\", \"A2\", \"B1\", \"B2\"]\r\n    observ_ids = [\"O1\"]\r\n    sample_metadata = [{}] * 4\r\n    test_table = biom.table.Table(data=test_data, observation_ids=observ_ids, sample_ids=sample_ids,\r\n                                  sample_metadata=sample_metadata, table_id=\"Example Table\")\r\n    sample_mapping = {\"A\": [\"A1\", \"A2\"], \"B\": [\"B1\", \"B2\"]}\r\n\r\nMy test failed, because for reasons beyond my understanding the first time `filtered_table.transform` calls my `transform_func` it doesn't really transform sample data in the table (that is `A1` remains intact), though `transform_func` itself works fine (because `B1` changes). Below is a demonstration from PyCharm debugging session\r\n\r\n![screen shot 2016-01-02 at 01 21 23](https://cloud.githubusercontent.com/assets/6906110/12072446/7beb1380-b0ef-11e5-8280-8600c0262640.png)\r\n\r\nIs this really a bug? \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/biocore/biom-format/issues/comments/168346754","html_url":"https://github.com/biocore/biom-format/issues/688#issuecomment-168346754","issue_url":"https://api.github.com/repos/biocore/biom-format/issues/688","id":168346754,"user":{"login":"wasade","id":474290,"avatar_url":"https://avatars.githubusercontent.com/u/474290?v=3","gravatar_id":"","url":"https://api.github.com/users/wasade","html_url":"https://github.com/wasade","followers_url":"https://api.github.com/users/wasade/followers","following_url":"https://api.github.com/users/wasade/following{/other_user}","gists_url":"https://api.github.com/users/wasade/gists{/gist_id}","starred_url":"https://api.github.com/users/wasade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wasade/subscriptions","organizations_url":"https://api.github.com/users/wasade/orgs","repos_url":"https://api.github.com/users/wasade/repos","events_url":"https://api.github.com/users/wasade/events{/privacy}","received_events_url":"https://api.github.com/users/wasade/received_events","type":"User","site_admin":false},"created_at":"2016-01-01T23:03:04Z","updated_at":"2016-01-01T23:03:04Z","body":"Can you simplify the example and provide expected output and observed\noutput? Im having a hard time determining what the issue is.\n\nOut of curiosity, does Table.collapse do what you need?\nOn Jan 1, 2016 2:24 PM, \"Eli\" <notifications@github.com> wrote:\n\n> def merge_samples(biom_table, sample_mapping):\n>     # TODO finalize docs. Stress that unlisted samples will be thrown away\n>     \"\"\"\n>\n>     :type biom_table: biom.table.Table\n>     :param sample_mapping: a dict that maps sample IDs to lists of IDs in\n>      biom_table to be merged (e.g. replicates), e.g. {merged_sample_id: [a set\n>      of old samples to merge]}\n>     :type sample_mapping: dict[str, collections.Sequence[str]]\n>     :return:\n>     \"\"\"\n>     def pick_temp_sample_id_and_invert(_sample_mapping):\n>         \"\"\"\n>         For each supersample pick the first listed subsample to use as a\n>         temporary ID and invert the mapping\n>         :type _sample_mapping: dict[str, collections.Sequence[str]]\n>         :param _sample_mapping: {merged_id: [a set of old samples to merge]}\n>         :rtype: dict[str, str]\n>         \"\"\"\n>         return {subsamples[0]: supersample for supersample, subsamples\n>                 in _sample_mapping.iteritems()}\n>\n>     def reduce_subsamples(_sample_mapping):\n>         \"\"\"\n>         Sum all subsamples of a supersample\n>         :type _sample_mapping: dict[str, collections.Sequence[str]]\n>         :rtype: dict[str, np.array]\n>         \"\"\"\n>         return {\n>             supersample: reduce(op.add,\n>                                 (biom_table.data(_id) for _id in subsamples))\n>             for supersample, subsamples in _sample_mapping.iteritems()\n>         }\n>\n>     # Merging subsamples. Picking temporary sample IDs and inverting the mapping\n>     # (required by `transform_func`) and dropping excess samples (i.e. that\n>     #  were merged)\n>     reduced_subsamples = reduce_subsamples(sample_mapping)\n>     temp_ids_to_new_ids = pick_temp_sample_id_and_invert(sample_mapping)\n>     ids_to_keep = temp_ids_to_new_ids.keys()\n>     filtered_table = biom_table.filter(ids_to_keep, inplace=False)\n>\n>     def transform_func(data, _id, metadata):\n>         \"\"\"\n>         Returns a reduced set of subsamples for each temporary _id\n>         :rtype: np.array\n>         \"\"\"\n>         return reduced_subsamples[temp_ids_to_new_ids[_id]]\n>\n>     # Transforming biom_table, i.e. replacing data of temp_ids with summed\n>     # arrays\n>     tranformed_biom = filtered_table.transform(transform_func, axis=\"sample\")\n>\n>     # Renaming merged samples\n>     return tranformed_biom.update_ids(temp_ids_to_new_ids)\n>\n> Here is the data I used for unit testing\n>\n> test_data = np.arange(4).reshape((1, 4))\n> sample_ids = [\"A1\", \"A2\", \"B1\", \"B2\"]\n> observ_ids = [\"O1\"]\n> sample_metadata = [{}] * 4\n> test_table = biom.table.Table(data=test_data, observation_ids=observ_ids, sample_ids=sample_ids,\n>                               sample_metadata=sample_metadata, table_id=\"Example Table\")\n> sample_mapping = {\"A\": [\"A1\", \"A2\"], \"B\": [\"B1\", \"B2\"]}\n>\n> My test failed, because for reasons beyond my understanding the first time\n> filtered_table.transform calls my transform_func it doesn't really\n> transform sample data in the table, though transform_func itself works\n> fine. Below is a demonstration from PyCharm debugging session\n>\n> [image: screen shot 2016-01-02 at 01 21 23]\n> <https://cloud.githubusercontent.com/assets/6906110/12072446/7beb1380-b0ef-11e5-8280-8600c0262640.png>\n>\n> Is this really a bug?\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/biocore/biom-format/issues/688>.\n>\n"}}