{"action":"created","comment":{"url":"https://api.github.com/repos/biocore/American-Gut/pulls/comments/42796136","id":42796136,"diff_hunk":"@@ -0,0 +1,72 @@\n+In this next chapter, we're going to calculate alpha diversity over all of the samples in the meta-analysis. As always, let's first get our environment setup.\n+\n+```python\n+>>> import os\n+>>> import multiprocessing\n+...\n+>>> import americangut.notebook_environment as agenv\n+>>> import americangut.util as agu\n+...\n+>>> import qiime_default_reference as qdr\n+...\n+>>> chp_path = agenv.activate('5')\n+>>> greengenes_tree = qdr.get_reference_tree()\n+>>> cpu_count = multiprocessing.cpu_count()\n+```\n+\n+And then we'll pull in the files we need for processing.\n+\n+```python\n+>>> ag_pgp_hmp_gg_100nt_biom = agu.get_existing_path(agenv.paths['ag-pgp-hmp-gg-100nt-biom'])\n+>>> ag_gg_100nt_biom         = agu.get_existing_path(agenv.paths['ag-gg-100nt-biom'])\n+>>> ag_pgp_hmp_gg_cleaned_md = agu.get_existing_path(agenv.paths['ag-pgp-hmp-gg-cleaned-md'])\n+```\n+\n+As well as establish the new paths that we'll be creating.\n+\n+```python\n+>>> ag_pgp_hmp_gg_100nt_1k_multi   = agu.get_new_path(agenv.paths['ag-pgp-hmp-gg-100nt-1k-multiple'])\n+...\n+>>> ag_pgp_hmp_gg_100nt_1k_adiv    = agu.get_new_path(agenv.paths['ag-pgp-hmp-gg-100nt-1k-adiv'])\n+>>> ag_pgp_hmp_gg_100nt_1k_adiv_pd = agu.get_new_path(agenv.paths['ag-pgp-hmp-gg-100nt-1k-adiv-pd'])\n+>>> ag_pgp_hmp_gg_100nt_1k_adiv_ch = agu.get_new_path(agenv.paths['ag-pgp-hmp-gg-100nt-1k-adiv-chao1'])\n+>>> ag_pgp_hmp_gg_100nt_1k_adiv_oo = agu.get_new_path(agenv.paths['ag-pgp-hmp-gg-100nt-1k-adiv-observedotus'])\n+```\n+\n+Now for the fun bits. We're going to make a call to `parallel_multiple_rarefactions.py` which will produce multiple tables normalized for sequencing effort at 1000 sequences per sample. This process is called rarefaction, and we perform it as the number of sequences produced per sample can vary quite a bit. This variation can greatly influence the resulting diversity scores, so in order to reduce bias driven by sequencing effort, we make an attempt to normalize. The process involves random subsampling, so we're going to do it a few times to get rough bounds on the diversity of each sample.","path":"ipynb/primary-processing/5-alpha_diversity.md","position":36,"original_position":36,"commit_id":"6bcd6ed74fb59fc9e879e5d0df96081d1902189a","original_commit_id":"6bcd6ed74fb59fc9e879e5d0df96081d1902189a","user":{"login":"mortonjt","id":4184797,"avatar_url":"https://avatars.githubusercontent.com/u/4184797?v=3","gravatar_id":"","url":"https://api.github.com/users/mortonjt","html_url":"https://github.com/mortonjt","followers_url":"https://api.github.com/users/mortonjt/followers","following_url":"https://api.github.com/users/mortonjt/following{/other_user}","gists_url":"https://api.github.com/users/mortonjt/gists{/gist_id}","starred_url":"https://api.github.com/users/mortonjt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mortonjt/subscriptions","organizations_url":"https://api.github.com/users/mortonjt/orgs","repos_url":"https://api.github.com/users/mortonjt/repos","events_url":"https://api.github.com/users/mortonjt/events{/privacy}","received_events_url":"https://api.github.com/users/mortonjt/received_events","type":"User","site_admin":false},"body":"Maybe specify that the tables are normalized via sampling without replacement?","created_at":"2015-10-22T19:42:14Z","updated_at":"2015-10-22T19:42:14Z","html_url":"https://github.com/biocore/American-Gut/pull/166#discussion_r42796136","pull_request_url":"https://api.github.com/repos/biocore/American-Gut/pulls/166","_links":{"self":{"href":"https://api.github.com/repos/biocore/American-Gut/pulls/comments/42796136"},"html":{"href":"https://github.com/biocore/American-Gut/pull/166#discussion_r42796136"},"pull_request":{"href":"https://api.github.com/repos/biocore/American-Gut/pulls/166"}}},"pull_request":{"url":"https://api.github.com/repos/biocore/American-Gut/pulls/166","id":48403805,"html_url":"https://github.com/biocore/American-Gut/pull/166","diff_url":"https://github.com/biocore/American-Gut/pull/166.diff","patch_url":"https://github.com/biocore/American-Gut/pull/166.patch","issue_url":"https://api.github.com/repos/biocore/American-Gut/issues/166","number":166,"state":"open","locked":false,"title":"5 alpha diversity","user":{"login":"wasade","id":474290,"avatar_url":"https://avatars.githubusercontent.com/u/474290?v=3","gravatar_id":"","url":"https://api.github.com/users/wasade","html_url":"https://github.com/wasade","followers_url":"https://api.github.com/users/wasade/followers","following_url":"https://api.github.com/users/wasade/following{/other_user}","gists_url":"https://api.github.com/users/wasade/gists{/gist_id}","starred_url":"https://api.github.com/users/wasade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wasade/subscriptions","organizations_url":"https://api.github.com/users/wasade/orgs","repos_url":"https://api.github.com/users/wasade/repos","events_url":"https://api.github.com/users/wasade/events{/privacy}","received_events_url":"https://api.github.com/users/wasade/received_events","type":"User","site_admin":false},"body":"This depends on #165, please do not review/merge until #165 has gone in.","created_at":"2015-10-22T00:03:57Z","updated_at":"2015-10-22T19:42:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"122f3a81706256a08d042086cd07d9b637dbef35","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/biocore/American-Gut/pulls/166/commits","review_comments_url":"https://api.github.com/repos/biocore/American-Gut/pulls/166/comments","review_comment_url":"https://api.github.com/repos/biocore/American-Gut/pulls/comments{/number}","comments_url":"https://api.github.com/repos/biocore/American-Gut/issues/166/comments","statuses_url":"https://api.github.com/repos/biocore/American-Gut/statuses/6bcd6ed74fb59fc9e879e5d0df96081d1902189a","head":{"label":"wasade:5-alpha_diversity","ref":"5-alpha_diversity","sha":"6bcd6ed74fb59fc9e879e5d0df96081d1902189a","user":{"login":"wasade","id":474290,"avatar_url":"https://avatars.githubusercontent.com/u/474290?v=3","gravatar_id":"","url":"https://api.github.com/users/wasade","html_url":"https://github.com/wasade","followers_url":"https://api.github.com/users/wasade/followers","following_url":"https://api.github.com/users/wasade/following{/other_user}","gists_url":"https://api.github.com/users/wasade/gists{/gist_id}","starred_url":"https://api.github.com/users/wasade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wasade/subscriptions","organizations_url":"https://api.github.com/users/wasade/orgs","repos_url":"https://api.github.com/users/wasade/repos","events_url":"https://api.github.com/users/wasade/events{/privacy}","received_events_url":"https://api.github.com/users/wasade/received_events","type":"User","site_admin":false},"repo":{"id":13258630,"name":"American-Gut","full_name":"wasade/American-Gut","owner":{"login":"wasade","id":474290,"avatar_url":"https://avatars.githubusercontent.com/u/474290?v=3","gravatar_id":"","url":"https://api.github.com/users/wasade","html_url":"https://github.com/wasade","followers_url":"https://api.github.com/users/wasade/followers","following_url":"https://api.github.com/users/wasade/following{/other_user}","gists_url":"https://api.github.com/users/wasade/gists{/gist_id}","starred_url":"https://api.github.com/users/wasade/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wasade/subscriptions","organizations_url":"https://api.github.com/users/wasade/orgs","repos_url":"https://api.github.com/users/wasade/repos","events_url":"https://api.github.com/users/wasade/events{/privacy}","received_events_url":"https://api.github.com/users/wasade/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/wasade/American-Gut","description":"American Gut open-access data and IPython notebooks","fork":true,"url":"https://api.github.com/repos/wasade/American-Gut","forks_url":"https://api.github.com/repos/wasade/American-Gut/forks","keys_url":"https://api.github.com/repos/wasade/American-Gut/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wasade/American-Gut/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wasade/American-Gut/teams","hooks_url":"https://api.github.com/repos/wasade/American-Gut/hooks","issue_events_url":"https://api.github.com/repos/wasade/American-Gut/issues/events{/number}","events_url":"https://api.github.com/repos/wasade/American-Gut/events","assignees_url":"https://api.github.com/repos/wasade/American-Gut/assignees{/user}","branches_url":"https://api.github.com/repos/wasade/American-Gut/branches{/branch}","tags_url":"https://api.github.com/repos/wasade/American-Gut/tags","blobs_url":"https://api.github.com/repos/wasade/American-Gut/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wasade/American-Gut/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wasade/American-Gut/git/refs{/sha}","trees_url":"https://api.github.com/repos/wasade/American-Gut/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wasade/American-Gut/statuses/{sha}","languages_url":"https://api.github.com/repos/wasade/American-Gut/languages","stargazers_url":"https://api.github.com/repos/wasade/American-Gut/stargazers","contributors_url":"https://api.github.com/repos/wasade/American-Gut/contributors","subscribers_url":"https://api.github.com/repos/wasade/American-Gut/subscribers","subscription_url":"https://api.github.com/repos/wasade/American-Gut/subscription","commits_url":"https://api.github.com/repos/wasade/American-Gut/commits{/sha}","git_commits_url":"https://api.github.com/repos/wasade/American-Gut/git/commits{/sha}","comments_url":"https://api.github.com/repos/wasade/American-Gut/comments{/number}","issue_comment_url":"https://api.github.com/repos/wasade/American-Gut/issues/comments{/number}","contents_url":"https://api.github.com/repos/wasade/American-Gut/contents/{+path}","compare_url":"https://api.github.com/repos/wasade/American-Gut/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wasade/American-Gut/merges","archive_url":"https://api.github.com/repos/wasade/American-Gut/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wasade/American-Gut/downloads","issues_url":"https://api.github.com/repos/wasade/American-Gut/issues{/number}","pulls_url":"https://api.github.com/repos/wasade/American-Gut/pulls{/number}","milestones_url":"https://api.github.com/repos/wasade/American-Gut/milestones{/number}","notifications_url":"https://api.github.com/repos/wasade/American-Gut/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wasade/American-Gut/labels{/name}","releases_url":"https://api.github.com/repos/wasade/American-Gut/releases{/id}","created_at":"2013-10-01T23:39:38Z","updated_at":"2015-03-12T15:14:56Z","pushed_at":"2015-10-22T19:40:05Z","git_url":"git://github.com/wasade/American-Gut.git","ssh_url":"git@github.com:wasade/American-Gut.git","clone_url":"https://github.com/wasade/American-Gut.git","svn_url":"https://github.com/wasade/American-Gut","homepage":null,"size":441568,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"biocore:master","ref":"master","sha":"75c61d922e0d740d069487fe19811a3d01fa6f7a","user":{"login":"biocore","id":6241857,"avatar_url":"https://avatars.githubusercontent.com/u/6241857?v=3","gravatar_id":"","url":"https://api.github.com/users/biocore","html_url":"https://github.com/biocore","followers_url":"https://api.github.com/users/biocore/followers","following_url":"https://api.github.com/users/biocore/following{/other_user}","gists_url":"https://api.github.com/users/biocore/gists{/gist_id}","starred_url":"https://api.github.com/users/biocore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/biocore/subscriptions","organizations_url":"https://api.github.com/users/biocore/orgs","repos_url":"https://api.github.com/users/biocore/repos","events_url":"https://api.github.com/users/biocore/events{/privacy}","received_events_url":"https://api.github.com/users/biocore/received_events","type":"Organization","site_admin":false},"repo":{"id":13258619,"name":"American-Gut","full_name":"biocore/American-Gut","owner":{"login":"biocore","id":6241857,"avatar_url":"https://avatars.githubusercontent.com/u/6241857?v=3","gravatar_id":"","url":"https://api.github.com/users/biocore","html_url":"https://github.com/biocore","followers_url":"https://api.github.com/users/biocore/followers","following_url":"https://api.github.com/users/biocore/following{/other_user}","gists_url":"https://api.github.com/users/biocore/gists{/gist_id}","starred_url":"https://api.github.com/users/biocore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/biocore/subscriptions","organizations_url":"https://api.github.com/users/biocore/orgs","repos_url":"https://api.github.com/users/biocore/repos","events_url":"https://api.github.com/users/biocore/events{/privacy}","received_events_url":"https://api.github.com/users/biocore/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/biocore/American-Gut","description":"American Gut open-access data and IPython notebooks","fork":false,"url":"https://api.github.com/repos/biocore/American-Gut","forks_url":"https://api.github.com/repos/biocore/American-Gut/forks","keys_url":"https://api.github.com/repos/biocore/American-Gut/keys{/key_id}","collaborators_url":"https://api.github.com/repos/biocore/American-Gut/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/biocore/American-Gut/teams","hooks_url":"https://api.github.com/repos/biocore/American-Gut/hooks","issue_events_url":"https://api.github.com/repos/biocore/American-Gut/issues/events{/number}","events_url":"https://api.github.com/repos/biocore/American-Gut/events","assignees_url":"https://api.github.com/repos/biocore/American-Gut/assignees{/user}","branches_url":"https://api.github.com/repos/biocore/American-Gut/branches{/branch}","tags_url":"https://api.github.com/repos/biocore/American-Gut/tags","blobs_url":"https://api.github.com/repos/biocore/American-Gut/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/biocore/American-Gut/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/biocore/American-Gut/git/refs{/sha}","trees_url":"https://api.github.com/repos/biocore/American-Gut/git/trees{/sha}","statuses_url":"https://api.github.com/repos/biocore/American-Gut/statuses/{sha}","languages_url":"https://api.github.com/repos/biocore/American-Gut/languages","stargazers_url":"https://api.github.com/repos/biocore/American-Gut/stargazers","contributors_url":"https://api.github.com/repos/biocore/American-Gut/contributors","subscribers_url":"https://api.github.com/repos/biocore/American-Gut/subscribers","subscription_url":"https://api.github.com/repos/biocore/American-Gut/subscription","commits_url":"https://api.github.com/repos/biocore/American-Gut/commits{/sha}","git_commits_url":"https://api.github.com/repos/biocore/American-Gut/git/commits{/sha}","comments_url":"https://api.github.com/repos/biocore/American-Gut/comments{/number}","issue_comment_url":"https://api.github.com/repos/biocore/American-Gut/issues/comments{/number}","contents_url":"https://api.github.com/repos/biocore/American-Gut/contents/{+path}","compare_url":"https://api.github.com/repos/biocore/American-Gut/compare/{base}...{head}","merges_url":"https://api.github.com/repos/biocore/American-Gut/merges","archive_url":"https://api.github.com/repos/biocore/American-Gut/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/biocore/American-Gut/downloads","issues_url":"https://api.github.com/repos/biocore/American-Gut/issues{/number}","pulls_url":"https://api.github.com/repos/biocore/American-Gut/pulls{/number}","milestones_url":"https://api.github.com/repos/biocore/American-Gut/milestones{/number}","notifications_url":"https://api.github.com/repos/biocore/American-Gut/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/biocore/American-Gut/labels{/name}","releases_url":"https://api.github.com/repos/biocore/American-Gut/releases{/id}","created_at":"2013-10-01T23:39:04Z","updated_at":"2015-07-31T07:31:44Z","pushed_at":"2015-10-22T19:40:35Z","git_url":"git://github.com/biocore/American-Gut.git","ssh_url":"git@github.com:biocore/American-Gut.git","clone_url":"https://github.com/biocore/American-Gut.git","svn_url":"https://github.com/biocore/American-Gut","homepage":null,"size":450845,"stargazers_count":23,"watchers_count":23,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":27,"mirror_url":null,"open_issues_count":22,"forks":27,"open_issues":22,"watchers":23,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/biocore/American-Gut/pulls/166"},"html":{"href":"https://github.com/biocore/American-Gut/pull/166"},"issue":{"href":"https://api.github.com/repos/biocore/American-Gut/issues/166"},"comments":{"href":"https://api.github.com/repos/biocore/American-Gut/issues/166/comments"},"review_comments":{"href":"https://api.github.com/repos/biocore/American-Gut/pulls/166/comments"},"review_comment":{"href":"https://api.github.com/repos/biocore/American-Gut/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/biocore/American-Gut/pulls/166/commits"},"statuses":{"href":"https://api.github.com/repos/biocore/American-Gut/statuses/6bcd6ed74fb59fc9e879e5d0df96081d1902189a"}}}}