{"action":"closed","number":10,"pull_request":{"url":"https://api.github.com/repos/pegler/pytzwhere/pulls/10","id":16717351,"html_url":"https://github.com/pegler/pytzwhere/pull/10","diff_url":"https://github.com/pegler/pytzwhere/pull/10.diff","patch_url":"https://github.com/pegler/pytzwhere/pull/10.patch","issue_url":"https://api.github.com/repos/pegler/pytzwhere/issues/10","number":10,"state":"closed","locked":false,"title":"Reduce memory consumption from 750MB to 60MB by using CSV not JSON. (Also lots of other changes.)","user":{"login":"gimbo","id":133770,"avatar_url":"https://avatars.githubusercontent.com/u/133770?v=3","gravatar_id":"","url":"https://api.github.com/users/gimbo","html_url":"https://github.com/gimbo","followers_url":"https://api.github.com/users/gimbo/followers","following_url":"https://api.github.com/users/gimbo/following{/other_user}","gists_url":"https://api.github.com/users/gimbo/gists{/gist_id}","starred_url":"https://api.github.com/users/gimbo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gimbo/subscriptions","organizations_url":"https://api.github.com/users/gimbo/orgs","repos_url":"https://api.github.com/users/gimbo/repos","events_url":"https://api.github.com/users/gimbo/events{/privacy}","received_events_url":"https://api.github.com/users/gimbo/received_events","type":"User","site_admin":false},"body":"Hi there. You might hate this, but in case you don't, please feel free to pull it and make a new release if you want. I've changed a *lot* though.\r\n\r\nThe main thing is that I noticed that reading in the JSON (before doing any processing) consumed 750MB of RAM - and actually this never got released properly ([apparently this is known behaviour with JSON and python](http://stackoverflow.com/a/11058050/392743)).  So I played around and found that by using a CSV file, each line of which is a single (tzname, polygon) pair, I can read that in one line at a time and save hundreds of MB.\r\n\r\nThen I found out that by representing points as (lat, lng) pairs rather than two-element dictionaries, we save a couple of hundred MB again.\r\n\r\nThen I found out that by using `numpy` arrays (if `numpy` is installed) instead of tuples for the polygons and lists of polygon lists, we can save even more.  (If `numpy` isn't installed, it falls back to tuples.)\r\n\r\nOverall, I've got the memory usage down from 750MB to 60MB in the best case.\r\n\r\nAlong the way I've considerably refactored the `tzwhere.py` script, including adding the capability to produce this CSV file, and also (this might be the bit you like least) moving from `argparse` to `docopt`, just because I find it more flexible.  That means the script's options have changed, and I have also changed the public interface of the `tzwhere` class, so this is quite a big change really, and potentially breaking for client code.\r\n\r\nWell, see what you think.  I'm happy to keep it as a separate fork if you want, but if you like it, great.\r\n\r\n(This is all with python2.7 however - not tested with anything else.)","created_at":"2014-06-04T16:03:25Z","updated_at":"2015-06-18T13:45:27Z","closed_at":"2015-06-18T13:45:27Z","merged_at":"2015-06-18T13:45:27Z","merge_commit_sha":"c0afd941f300e879fa700c8a0d60ec9ce1764889","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/pegler/pytzwhere/pulls/10/commits","review_comments_url":"https://api.github.com/repos/pegler/pytzwhere/pulls/10/comments","review_comment_url":"https://api.github.com/repos/pegler/pytzwhere/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pegler/pytzwhere/issues/10/comments","statuses_url":"https://api.github.com/repos/pegler/pytzwhere/statuses/b4edc858d542a62bee542d38e14f5d78c5aaa441","head":{"label":"gimbo:master","ref":"master","sha":"b4edc858d542a62bee542d38e14f5d78c5aaa441","user":{"login":"gimbo","id":133770,"avatar_url":"https://avatars.githubusercontent.com/u/133770?v=3","gravatar_id":"","url":"https://api.github.com/users/gimbo","html_url":"https://github.com/gimbo","followers_url":"https://api.github.com/users/gimbo/followers","following_url":"https://api.github.com/users/gimbo/following{/other_user}","gists_url":"https://api.github.com/users/gimbo/gists{/gist_id}","starred_url":"https://api.github.com/users/gimbo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gimbo/subscriptions","organizations_url":"https://api.github.com/users/gimbo/orgs","repos_url":"https://api.github.com/users/gimbo/repos","events_url":"https://api.github.com/users/gimbo/events{/privacy}","received_events_url":"https://api.github.com/users/gimbo/received_events","type":"User","site_admin":false},"repo":{"id":20480305,"name":"pytzwhere","full_name":"gimbo/pytzwhere","owner":{"login":"gimbo","id":133770,"avatar_url":"https://avatars.githubusercontent.com/u/133770?v=3","gravatar_id":"","url":"https://api.github.com/users/gimbo","html_url":"https://github.com/gimbo","followers_url":"https://api.github.com/users/gimbo/followers","following_url":"https://api.github.com/users/gimbo/following{/other_user}","gists_url":"https://api.github.com/users/gimbo/gists{/gist_id}","starred_url":"https://api.github.com/users/gimbo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gimbo/subscriptions","organizations_url":"https://api.github.com/users/gimbo/orgs","repos_url":"https://api.github.com/users/gimbo/repos","events_url":"https://api.github.com/users/gimbo/events{/privacy}","received_events_url":"https://api.github.com/users/gimbo/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/gimbo/pytzwhere","description":"python library to look up timezone from lat lng offline","fork":true,"url":"https://api.github.com/repos/gimbo/pytzwhere","forks_url":"https://api.github.com/repos/gimbo/pytzwhere/forks","keys_url":"https://api.github.com/repos/gimbo/pytzwhere/keys{/key_id}","collaborators_url":"https://api.github.com/repos/gimbo/pytzwhere/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/gimbo/pytzwhere/teams","hooks_url":"https://api.github.com/repos/gimbo/pytzwhere/hooks","issue_events_url":"https://api.github.com/repos/gimbo/pytzwhere/issues/events{/number}","events_url":"https://api.github.com/repos/gimbo/pytzwhere/events","assignees_url":"https://api.github.com/repos/gimbo/pytzwhere/assignees{/user}","branches_url":"https://api.github.com/repos/gimbo/pytzwhere/branches{/branch}","tags_url":"https://api.github.com/repos/gimbo/pytzwhere/tags","blobs_url":"https://api.github.com/repos/gimbo/pytzwhere/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/gimbo/pytzwhere/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/gimbo/pytzwhere/git/refs{/sha}","trees_url":"https://api.github.com/repos/gimbo/pytzwhere/git/trees{/sha}","statuses_url":"https://api.github.com/repos/gimbo/pytzwhere/statuses/{sha}","languages_url":"https://api.github.com/repos/gimbo/pytzwhere/languages","stargazers_url":"https://api.github.com/repos/gimbo/pytzwhere/stargazers","contributors_url":"https://api.github.com/repos/gimbo/pytzwhere/contributors","subscribers_url":"https://api.github.com/repos/gimbo/pytzwhere/subscribers","subscription_url":"https://api.github.com/repos/gimbo/pytzwhere/subscription","commits_url":"https://api.github.com/repos/gimbo/pytzwhere/commits{/sha}","git_commits_url":"https://api.github.com/repos/gimbo/pytzwhere/git/commits{/sha}","comments_url":"https://api.github.com/repos/gimbo/pytzwhere/comments{/number}","issue_comment_url":"https://api.github.com/repos/gimbo/pytzwhere/issues/comments{/number}","contents_url":"https://api.github.com/repos/gimbo/pytzwhere/contents/{+path}","compare_url":"https://api.github.com/repos/gimbo/pytzwhere/compare/{base}...{head}","merges_url":"https://api.github.com/repos/gimbo/pytzwhere/merges","archive_url":"https://api.github.com/repos/gimbo/pytzwhere/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/gimbo/pytzwhere/downloads","issues_url":"https://api.github.com/repos/gimbo/pytzwhere/issues{/number}","pulls_url":"https://api.github.com/repos/gimbo/pytzwhere/pulls{/number}","milestones_url":"https://api.github.com/repos/gimbo/pytzwhere/milestones{/number}","notifications_url":"https://api.github.com/repos/gimbo/pytzwhere/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/gimbo/pytzwhere/labels{/name}","releases_url":"https://api.github.com/repos/gimbo/pytzwhere/releases{/id}","created_at":"2014-06-04T11:14:05Z","updated_at":"2014-12-16T18:16:13Z","pushed_at":"2014-12-16T18:16:12Z","git_url":"git://github.com/gimbo/pytzwhere.git","ssh_url":"git@github.com:gimbo/pytzwhere.git","clone_url":"https://github.com/gimbo/pytzwhere.git","svn_url":"https://github.com/gimbo/pytzwhere","homepage":null,"size":68971,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"open_issues_count":0,"forks":5,"open_issues":0,"watchers":2,"default_branch":"master"}},"base":{"label":"pegler:master","ref":"master","sha":"cf4909b6bdfe93502b2cd402e9e6048de51a47a7","user":{"login":"pegler","id":94491,"avatar_url":"https://avatars.githubusercontent.com/u/94491?v=3","gravatar_id":"","url":"https://api.github.com/users/pegler","html_url":"https://github.com/pegler","followers_url":"https://api.github.com/users/pegler/followers","following_url":"https://api.github.com/users/pegler/following{/other_user}","gists_url":"https://api.github.com/users/pegler/gists{/gist_id}","starred_url":"https://api.github.com/users/pegler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pegler/subscriptions","organizations_url":"https://api.github.com/users/pegler/orgs","repos_url":"https://api.github.com/users/pegler/repos","events_url":"https://api.github.com/users/pegler/events{/privacy}","received_events_url":"https://api.github.com/users/pegler/received_events","type":"User","site_admin":false},"repo":{"id":5545325,"name":"pytzwhere","full_name":"pegler/pytzwhere","owner":{"login":"pegler","id":94491,"avatar_url":"https://avatars.githubusercontent.com/u/94491?v=3","gravatar_id":"","url":"https://api.github.com/users/pegler","html_url":"https://github.com/pegler","followers_url":"https://api.github.com/users/pegler/followers","following_url":"https://api.github.com/users/pegler/following{/other_user}","gists_url":"https://api.github.com/users/pegler/gists{/gist_id}","starred_url":"https://api.github.com/users/pegler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pegler/subscriptions","organizations_url":"https://api.github.com/users/pegler/orgs","repos_url":"https://api.github.com/users/pegler/repos","events_url":"https://api.github.com/users/pegler/events{/privacy}","received_events_url":"https://api.github.com/users/pegler/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pegler/pytzwhere","description":"python library to look up timezone from lat lng offline","fork":false,"url":"https://api.github.com/repos/pegler/pytzwhere","forks_url":"https://api.github.com/repos/pegler/pytzwhere/forks","keys_url":"https://api.github.com/repos/pegler/pytzwhere/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pegler/pytzwhere/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pegler/pytzwhere/teams","hooks_url":"https://api.github.com/repos/pegler/pytzwhere/hooks","issue_events_url":"https://api.github.com/repos/pegler/pytzwhere/issues/events{/number}","events_url":"https://api.github.com/repos/pegler/pytzwhere/events","assignees_url":"https://api.github.com/repos/pegler/pytzwhere/assignees{/user}","branches_url":"https://api.github.com/repos/pegler/pytzwhere/branches{/branch}","tags_url":"https://api.github.com/repos/pegler/pytzwhere/tags","blobs_url":"https://api.github.com/repos/pegler/pytzwhere/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pegler/pytzwhere/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pegler/pytzwhere/git/refs{/sha}","trees_url":"https://api.github.com/repos/pegler/pytzwhere/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pegler/pytzwhere/statuses/{sha}","languages_url":"https://api.github.com/repos/pegler/pytzwhere/languages","stargazers_url":"https://api.github.com/repos/pegler/pytzwhere/stargazers","contributors_url":"https://api.github.com/repos/pegler/pytzwhere/contributors","subscribers_url":"https://api.github.com/repos/pegler/pytzwhere/subscribers","subscription_url":"https://api.github.com/repos/pegler/pytzwhere/subscription","commits_url":"https://api.github.com/repos/pegler/pytzwhere/commits{/sha}","git_commits_url":"https://api.github.com/repos/pegler/pytzwhere/git/commits{/sha}","comments_url":"https://api.github.com/repos/pegler/pytzwhere/comments{/number}","issue_comment_url":"https://api.github.com/repos/pegler/pytzwhere/issues/comments{/number}","contents_url":"https://api.github.com/repos/pegler/pytzwhere/contents/{+path}","compare_url":"https://api.github.com/repos/pegler/pytzwhere/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pegler/pytzwhere/merges","archive_url":"https://api.github.com/repos/pegler/pytzwhere/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pegler/pytzwhere/downloads","issues_url":"https://api.github.com/repos/pegler/pytzwhere/issues{/number}","pulls_url":"https://api.github.com/repos/pegler/pytzwhere/pulls{/number}","milestones_url":"https://api.github.com/repos/pegler/pytzwhere/milestones{/number}","notifications_url":"https://api.github.com/repos/pegler/pytzwhere/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pegler/pytzwhere/labels{/name}","releases_url":"https://api.github.com/repos/pegler/pytzwhere/releases{/id}","created_at":"2012-08-24T20:15:27Z","updated_at":"2015-05-26T23:08:31Z","pushed_at":"2015-06-18T13:45:27Z","git_url":"git://github.com/pegler/pytzwhere.git","ssh_url":"git@github.com:pegler/pytzwhere.git","clone_url":"https://github.com/pegler/pytzwhere.git","svn_url":"https://github.com/pegler/pytzwhere","homepage":null,"size":69019,"stargazers_count":32,"watchers_count":32,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":28,"mirror_url":null,"open_issues_count":9,"forks":28,"open_issues":9,"watchers":32,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pegler/pytzwhere/pulls/10"},"html":{"href":"https://github.com/pegler/pytzwhere/pull/10"},"issue":{"href":"https://api.github.com/repos/pegler/pytzwhere/issues/10"},"comments":{"href":"https://api.github.com/repos/pegler/pytzwhere/issues/10/comments"},"review_comments":{"href":"https://api.github.com/repos/pegler/pytzwhere/pulls/10/comments"},"review_comment":{"href":"https://api.github.com/repos/pegler/pytzwhere/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pegler/pytzwhere/pulls/10/commits"},"statuses":{"href":"https://api.github.com/repos/pegler/pytzwhere/statuses/b4edc858d542a62bee542d38e14f5d78c5aaa441"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"cstich","id":9212959,"avatar_url":"https://avatars.githubusercontent.com/u/9212959?v=3","gravatar_id":"","url":"https://api.github.com/users/cstich","html_url":"https://github.com/cstich","followers_url":"https://api.github.com/users/cstich/followers","following_url":"https://api.github.com/users/cstich/following{/other_user}","gists_url":"https://api.github.com/users/cstich/gists{/gist_id}","starred_url":"https://api.github.com/users/cstich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cstich/subscriptions","organizations_url":"https://api.github.com/users/cstich/orgs","repos_url":"https://api.github.com/users/cstich/repos","events_url":"https://api.github.com/users/cstich/events{/privacy}","received_events_url":"https://api.github.com/users/cstich/received_events","type":"User","site_admin":false},"comments":13,"review_comments":0,"commits":17,"additions":28117,"deletions":95,"changed_files":4}}