{"action":"opened","number":407,"pull_request":{"url":"https://api.github.com/repos/arq5x/gemini/pulls/407","id":31838871,"html_url":"https://github.com/arq5x/gemini/pull/407","diff_url":"https://github.com/arq5x/gemini/pull/407.diff","patch_url":"https://github.com/arq5x/gemini/pull/407.patch","issue_url":"https://api.github.com/repos/arq5x/gemini/issues/407","number":407,"state":"open","locked":false,"title":"WIP: improve load performance","user":{"login":"brentp","id":1739,"avatar_url":"https://avatars.githubusercontent.com/u/1739?v=3","gravatar_id":"","url":"https://api.github.com/users/brentp","html_url":"https://github.com/brentp","followers_url":"https://api.github.com/users/brentp/followers","following_url":"https://api.github.com/users/brentp/following{/other_user}","gists_url":"https://api.github.com/users/brentp/gists{/gist_id}","starred_url":"https://api.github.com/users/brentp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brentp/subscriptions","organizations_url":"https://api.github.com/users/brentp/orgs","repos_url":"https://api.github.com/users/brentp/repos","events_url":"https://api.github.com/users/brentp/events{/privacy}","received_events_url":"https://api.github.com/users/brentp/received_events","type":"User","site_admin":false},"body":"Work in progress...\r\n\r\nOn master, the load-time is dominated by stuff related to bigwig (even without GERP). For loading about 10K records on a slow laptop we get:\r\n\r\n```\r\n         201859321 function calls (201853743 primitive calls) in 816.745 seconds\r\n\r\n   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n 48414395  229.792    0.000  371.800    0.000 binary_file.py:51(read_and_unpack)\r\n 32591931   96.148    0.000  346.071    0.000 binary_file.py:98(read_uint32)\r\n     9865   87.024    0.009  603.337    0.061 {method 'summarize' of 'bx.bbi.bbi_file.BBIFile' objects}\r\n 48444676   68.272    0.000   68.272    0.000 {_struct.unpack}\r\n        6   52.071    8.679   52.071    8.679 {method 'executemany' of 'sqlite3.Cursor' objects}\r\n 30372230   43.242    0.000   43.242    0.000 {method 'read' of 'cStringIO.StringI' objects}\r\n 18086005   30.800    0.000   30.800    0.000 {method 'read' of 'file' objects}\r\n 10100736   29.896    0.000  106.517    0.000 binary_file.py:104(read_float)\r\n     9865   23.537    0.002   24.426    0.002 annotations.py:639(get_exac_info)\r\n  5613211   16.796    0.000   61.063    0.000 binary_file.py:101(read_uint64)\r\n       78   14.811    0.190   14.811    0.190 {method 'execute' of 'sqlite3.Cursor' objects}\r\n        1    9.812    9.812   24.710   24.710 gemini_load_chunk.py:583(_get_gene_detailed)\r\n     9865    7.046    0.001    7.618    0.001 annotations.py:613(get_1000G_info)\r\n     9865    6.019    0.001    6.475    0.001 annotations.py:784(get_recomb_info)\r\n     9865    5.607    0.001    6.056    0.001 annotations.py:546(get_dbsnp_info)\r\n```\r\n\r\nWith the changes in the first commit here, to use bed.gz and tabix instead of bigwig, this goes to:\r\n\r\n```\r\n         7955140 function calls (7949562 primitive calls) in 274.318 seconds\r\n\r\n   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n     9865   60.558    0.006   60.968    0.006 annotations.py:390(get_fitcons)\r\n        6   53.082    8.847   53.082    8.847 {method 'executemany' of 'sqlite3.Cursor' objects}\r\n     9865   22.795    0.002   23.404    0.002 annotations.py:635(get_exac_info)\r\n       78   14.932    0.191   14.932    0.191 {method 'execute' of 'sqlite3.Cursor' objects}\r\n        1    9.952    9.952   25.312   25.312 gemini_load_chunk.py:583(_get_gene_detailed)\r\n     9865    7.178    0.001    7.750    0.001 annotations.py:609(get_1000G_info)\r\n     9865    5.979    0.001    6.426    0.001 annotations.py:780(get_recomb_info)\r\n     9865    5.831    0.001    6.279    0.001 annotations.py:542(get_dbsnp_info)\r\n     9865    5.055    0.001    5.070    0.001 annotations.py:796(_get_first_vcf_hit)\r\n     9865    5.051    0.001    5.550    0.001 annotations.py:876(get_encode_consensus_segs)\r\n     9865    4.950    0.001    5.397    0.001 annotations.py:367(get_gerp_elements)\r\n     9865    4.500    0.000    4.942    0.001 annotations.py:740(get_rmsk_info)\r\n     9865    4.339    0.000    4.801    0.000 annotations.py:838(get_encode_tfbs)\r\n     9865    4.332    0.000    4.728    0.000 annotations.py:761(get_conservation_info)\r\n     9865    4.274    0.000    4.651    0.000 annotations.py:831(get_cse)\r\n```\r\n\r\nSo, **274** seconds instead of **817**. This should be a general speed improvement for all loads on the order of 3X. \r\n","created_at":"2015-03-24T15:42:05Z","updated_at":"2015-03-24T15:42:05Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/arq5x/gemini/pulls/407/commits","review_comments_url":"https://api.github.com/repos/arq5x/gemini/pulls/407/comments","review_comment_url":"https://api.github.com/repos/arq5x/gemini/pulls/comments{/number}","comments_url":"https://api.github.com/repos/arq5x/gemini/issues/407/comments","statuses_url":"https://api.github.com/repos/arq5x/gemini/statuses/b8571c9c4ab0976cd0b736dbc5d9454254844b4c","head":{"label":"brentp:load-perf","ref":"load-perf","sha":"b8571c9c4ab0976cd0b736dbc5d9454254844b4c","user":{"login":"brentp","id":1739,"avatar_url":"https://avatars.githubusercontent.com/u/1739?v=3","gravatar_id":"","url":"https://api.github.com/users/brentp","html_url":"https://github.com/brentp","followers_url":"https://api.github.com/users/brentp/followers","following_url":"https://api.github.com/users/brentp/following{/other_user}","gists_url":"https://api.github.com/users/brentp/gists{/gist_id}","starred_url":"https://api.github.com/users/brentp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brentp/subscriptions","organizations_url":"https://api.github.com/users/brentp/orgs","repos_url":"https://api.github.com/users/brentp/repos","events_url":"https://api.github.com/users/brentp/events{/privacy}","received_events_url":"https://api.github.com/users/brentp/received_events","type":"User","site_admin":false},"repo":{"id":31544063,"name":"gemini","full_name":"brentp/gemini","owner":{"login":"brentp","id":1739,"avatar_url":"https://avatars.githubusercontent.com/u/1739?v=3","gravatar_id":"","url":"https://api.github.com/users/brentp","html_url":"https://github.com/brentp","followers_url":"https://api.github.com/users/brentp/followers","following_url":"https://api.github.com/users/brentp/following{/other_user}","gists_url":"https://api.github.com/users/brentp/gists{/gist_id}","starred_url":"https://api.github.com/users/brentp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brentp/subscriptions","organizations_url":"https://api.github.com/users/brentp/orgs","repos_url":"https://api.github.com/users/brentp/repos","events_url":"https://api.github.com/users/brentp/events{/privacy}","received_events_url":"https://api.github.com/users/brentp/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/brentp/gemini","description":"a lightweight db framework for disease and population genetics.","fork":true,"url":"https://api.github.com/repos/brentp/gemini","forks_url":"https://api.github.com/repos/brentp/gemini/forks","keys_url":"https://api.github.com/repos/brentp/gemini/keys{/key_id}","collaborators_url":"https://api.github.com/repos/brentp/gemini/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/brentp/gemini/teams","hooks_url":"https://api.github.com/repos/brentp/gemini/hooks","issue_events_url":"https://api.github.com/repos/brentp/gemini/issues/events{/number}","events_url":"https://api.github.com/repos/brentp/gemini/events","assignees_url":"https://api.github.com/repos/brentp/gemini/assignees{/user}","branches_url":"https://api.github.com/repos/brentp/gemini/branches{/branch}","tags_url":"https://api.github.com/repos/brentp/gemini/tags","blobs_url":"https://api.github.com/repos/brentp/gemini/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/brentp/gemini/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/brentp/gemini/git/refs{/sha}","trees_url":"https://api.github.com/repos/brentp/gemini/git/trees{/sha}","statuses_url":"https://api.github.com/repos/brentp/gemini/statuses/{sha}","languages_url":"https://api.github.com/repos/brentp/gemini/languages","stargazers_url":"https://api.github.com/repos/brentp/gemini/stargazers","contributors_url":"https://api.github.com/repos/brentp/gemini/contributors","subscribers_url":"https://api.github.com/repos/brentp/gemini/subscribers","subscription_url":"https://api.github.com/repos/brentp/gemini/subscription","commits_url":"https://api.github.com/repos/brentp/gemini/commits{/sha}","git_commits_url":"https://api.github.com/repos/brentp/gemini/git/commits{/sha}","comments_url":"https://api.github.com/repos/brentp/gemini/comments{/number}","issue_comment_url":"https://api.github.com/repos/brentp/gemini/issues/comments{/number}","contents_url":"https://api.github.com/repos/brentp/gemini/contents/{+path}","compare_url":"https://api.github.com/repos/brentp/gemini/compare/{base}...{head}","merges_url":"https://api.github.com/repos/brentp/gemini/merges","archive_url":"https://api.github.com/repos/brentp/gemini/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/brentp/gemini/downloads","issues_url":"https://api.github.com/repos/brentp/gemini/issues{/number}","pulls_url":"https://api.github.com/repos/brentp/gemini/pulls{/number}","milestones_url":"https://api.github.com/repos/brentp/gemini/milestones{/number}","notifications_url":"https://api.github.com/repos/brentp/gemini/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/brentp/gemini/labels{/name}","releases_url":"https://api.github.com/repos/brentp/gemini/releases{/id}","created_at":"2015-03-02T14:28:43Z","updated_at":"2015-03-23T16:53:27Z","pushed_at":"2015-03-24T15:36:09Z","git_url":"git://github.com/brentp/gemini.git","ssh_url":"git@github.com:brentp/gemini.git","clone_url":"https://github.com/brentp/gemini.git","svn_url":"https://github.com/brentp/gemini","homepage":"","size":53444,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"arq5x:master","ref":"master","sha":"4b36ae53b114005fea5c592b448c33b39c9d8d13","user":{"login":"arq5x","id":72291,"avatar_url":"https://avatars.githubusercontent.com/u/72291?v=3","gravatar_id":"","url":"https://api.github.com/users/arq5x","html_url":"https://github.com/arq5x","followers_url":"https://api.github.com/users/arq5x/followers","following_url":"https://api.github.com/users/arq5x/following{/other_user}","gists_url":"https://api.github.com/users/arq5x/gists{/gist_id}","starred_url":"https://api.github.com/users/arq5x/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arq5x/subscriptions","organizations_url":"https://api.github.com/users/arq5x/orgs","repos_url":"https://api.github.com/users/arq5x/repos","events_url":"https://api.github.com/users/arq5x/events{/privacy}","received_events_url":"https://api.github.com/users/arq5x/received_events","type":"User","site_admin":false},"repo":{"id":3730529,"name":"gemini","full_name":"arq5x/gemini","owner":{"login":"arq5x","id":72291,"avatar_url":"https://avatars.githubusercontent.com/u/72291?v=3","gravatar_id":"","url":"https://api.github.com/users/arq5x","html_url":"https://github.com/arq5x","followers_url":"https://api.github.com/users/arq5x/followers","following_url":"https://api.github.com/users/arq5x/following{/other_user}","gists_url":"https://api.github.com/users/arq5x/gists{/gist_id}","starred_url":"https://api.github.com/users/arq5x/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arq5x/subscriptions","organizations_url":"https://api.github.com/users/arq5x/orgs","repos_url":"https://api.github.com/users/arq5x/repos","events_url":"https://api.github.com/users/arq5x/events{/privacy}","received_events_url":"https://api.github.com/users/arq5x/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/arq5x/gemini","description":"a lightweight db framework for disease and population genetics.","fork":false,"url":"https://api.github.com/repos/arq5x/gemini","forks_url":"https://api.github.com/repos/arq5x/gemini/forks","keys_url":"https://api.github.com/repos/arq5x/gemini/keys{/key_id}","collaborators_url":"https://api.github.com/repos/arq5x/gemini/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/arq5x/gemini/teams","hooks_url":"https://api.github.com/repos/arq5x/gemini/hooks","issue_events_url":"https://api.github.com/repos/arq5x/gemini/issues/events{/number}","events_url":"https://api.github.com/repos/arq5x/gemini/events","assignees_url":"https://api.github.com/repos/arq5x/gemini/assignees{/user}","branches_url":"https://api.github.com/repos/arq5x/gemini/branches{/branch}","tags_url":"https://api.github.com/repos/arq5x/gemini/tags","blobs_url":"https://api.github.com/repos/arq5x/gemini/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/arq5x/gemini/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/arq5x/gemini/git/refs{/sha}","trees_url":"https://api.github.com/repos/arq5x/gemini/git/trees{/sha}","statuses_url":"https://api.github.com/repos/arq5x/gemini/statuses/{sha}","languages_url":"https://api.github.com/repos/arq5x/gemini/languages","stargazers_url":"https://api.github.com/repos/arq5x/gemini/stargazers","contributors_url":"https://api.github.com/repos/arq5x/gemini/contributors","subscribers_url":"https://api.github.com/repos/arq5x/gemini/subscribers","subscription_url":"https://api.github.com/repos/arq5x/gemini/subscription","commits_url":"https://api.github.com/repos/arq5x/gemini/commits{/sha}","git_commits_url":"https://api.github.com/repos/arq5x/gemini/git/commits{/sha}","comments_url":"https://api.github.com/repos/arq5x/gemini/comments{/number}","issue_comment_url":"https://api.github.com/repos/arq5x/gemini/issues/comments{/number}","contents_url":"https://api.github.com/repos/arq5x/gemini/contents/{+path}","compare_url":"https://api.github.com/repos/arq5x/gemini/compare/{base}...{head}","merges_url":"https://api.github.com/repos/arq5x/gemini/merges","archive_url":"https://api.github.com/repos/arq5x/gemini/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/arq5x/gemini/downloads","issues_url":"https://api.github.com/repos/arq5x/gemini/issues{/number}","pulls_url":"https://api.github.com/repos/arq5x/gemini/pulls{/number}","milestones_url":"https://api.github.com/repos/arq5x/gemini/milestones{/number}","notifications_url":"https://api.github.com/repos/arq5x/gemini/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/arq5x/gemini/labels{/name}","releases_url":"https://api.github.com/repos/arq5x/gemini/releases{/id}","created_at":"2012-03-15T16:55:29Z","updated_at":"2015-03-24T13:28:22Z","pushed_at":"2015-03-24T13:28:22Z","git_url":"git://github.com/arq5x/gemini.git","ssh_url":"git@github.com:arq5x/gemini.git","clone_url":"https://github.com/arq5x/gemini.git","svn_url":"https://github.com/arq5x/gemini","homepage":"","size":55987,"stargazers_count":99,"watchers_count":99,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":49,"mirror_url":null,"open_issues_count":66,"forks":49,"open_issues":66,"watchers":99,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/arq5x/gemini/pulls/407"},"html":{"href":"https://github.com/arq5x/gemini/pull/407"},"issue":{"href":"https://api.github.com/repos/arq5x/gemini/issues/407"},"comments":{"href":"https://api.github.com/repos/arq5x/gemini/issues/407/comments"},"review_comments":{"href":"https://api.github.com/repos/arq5x/gemini/pulls/407/comments"},"review_comment":{"href":"https://api.github.com/repos/arq5x/gemini/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/arq5x/gemini/pulls/407/commits"},"statuses":{"href":"https://api.github.com/repos/arq5x/gemini/statuses/b8571c9c4ab0976cd0b736dbc5d9454254844b4c"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":25,"deletions":24,"changed_files":2}}