{"action":"created","comment":{"url":"https://api.github.com/repos/pelias/openstreetmap/pulls/comments/26114457","id":26114457,"diff_hunk":"@@ -5,92 +5,47 @@ var stats = require('../../stream/stats'),\n module.exports.tests = {};","path":"test/stream/stats.js","position":1,"original_position":1,"commit_id":"810f23564be4862108e5d57ba6ef05a9a0f6e0f7","original_commit_id":"810f23564be4862108e5d57ba6ef05a9a0f6e0f7","user":{"login":"missinglink","id":738069,"avatar_url":"https://avatars.githubusercontent.com/u/738069?v=3","gravatar_id":"","url":"https://api.github.com/users/missinglink","html_url":"https://github.com/missinglink","followers_url":"https://api.github.com/users/missinglink/followers","following_url":"https://api.github.com/users/missinglink/following{/other_user}","gists_url":"https://api.github.com/users/missinglink/gists{/gist_id}","starred_url":"https://api.github.com/users/missinglink/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/missinglink/subscriptions","organizations_url":"https://api.github.com/users/missinglink/orgs","repos_url":"https://api.github.com/users/missinglink/repos","events_url":"https://api.github.com/users/missinglink/events{/privacy}","received_events_url":"https://api.github.com/users/missinglink/received_events","type":"User","site_admin":false},"body":"It could be moved to an external module, I will open a ticket for this and the other tasks to externalise functionality not in the osm domain.","created_at":"2015-03-10T11:41:14Z","updated_at":"2015-03-10T11:41:14Z","html_url":"https://github.com/pelias/openstreetmap/pull/27#discussion_r26114457","pull_request_url":"https://api.github.com/repos/pelias/openstreetmap/pulls/27","_links":{"self":{"href":"https://api.github.com/repos/pelias/openstreetmap/pulls/comments/26114457"},"html":{"href":"https://github.com/pelias/openstreetmap/pull/27#discussion_r26114457"},"pull_request":{"href":"https://api.github.com/repos/pelias/openstreetmap/pulls/27"}}},"pull_request":{"url":"https://api.github.com/repos/pelias/openstreetmap/pulls/27","id":25602113,"html_url":"https://github.com/pelias/openstreetmap/pull/27","diff_url":"https://github.com/pelias/openstreetmap/pull/27.diff","patch_url":"https://github.com/pelias/openstreetmap/pull/27.patch","issue_url":"https://api.github.com/repos/pelias/openstreetmap/issues/27","number":27,"state":"open","locked":false,"title":"openstreemap pipeline improvements","user":{"login":"missinglink","id":738069,"avatar_url":"https://avatars.githubusercontent.com/u/738069?v=3","gravatar_id":"","url":"https://api.github.com/users/missinglink","html_url":"https://github.com/missinglink","followers_url":"https://api.github.com/users/missinglink/followers","following_url":"https://api.github.com/users/missinglink/following{/other_user}","gists_url":"https://api.github.com/users/missinglink/gists{/gist_id}","starred_url":"https://api.github.com/users/missinglink/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/missinglink/subscriptions","organizations_url":"https://api.github.com/users/missinglink/orgs","repos_url":"https://api.github.com/users/missinglink/repos","events_url":"https://api.github.com/users/missinglink/events{/privacy}","received_events_url":"https://api.github.com/users/missinglink/received_events","type":"User","site_admin":false},"body":"PENDING REVIEW\r\n\r\nFirst of all; apologies for the very large PR. This work is the result of research and development of the best way to build the OSM import pipeline over the last 3-4 months.\r\n\r\nThe result is a far more refined, stable and tested import pipeline which we can continue to build off going-forward.\r\n\r\nThe current master branch (the target of this PR) has 4 major pain points:\r\n\r\n1. `slow imports` - it can take up to 20 days to import the full planet file, it's not multi-core.\r\n2. `failure to exit(0)` - sometimes the imports fail to exit cleanly, making ops scripting difficult.\r\n3. ` Recursive process.nextTick detected` - streams-related errors.\r\n4. `code clarity/testing` - the code is only partially tested and in some places, it's messy. \r\n\r\nIn order to address [1] the following work was done, which resulted in drastic speed improvements:\r\n\r\n- [x] implement `golang` pbf parser \r\n- [x] implement admin-lookup\r\n- [x] upgrade to the latest suggester-pipeline module\r\n- [x] simplify features.js\r\n- [x] remove osm_types.js\r\n\r\nIn order to fix [2], I simply:\r\n\r\n- [x] fix stats.js module via unit tests, remove complexity\r\n\r\nIssue [3] was resolved by fixing the above issues and:\r\n\r\n- [x] upgrade all dependencies.\r\n- [x] fix nodejs version inconsistencies to work with both `0.10` and `0.12`.\r\n\r\nIssue [4] was tackled by:\r\n\r\n- [x] implement pelias-model\r\n- [x] add doc.setAddress() and doc.getAddress() functions to pelias/model\r\n- [x] general clean up\r\n- [x] improve global module.exports, add tests.\r\n- [x] configure code linting and precommit hook, lint everything.\r\n- [x] make osm data mappers clearer and easier to write/modify\r\n- [x] add a unit test for every module\r\n- [x] add detailed comments at the top of complex streams to explain their purpose\r\n- [x] add end-to-end system test\r\n\r\nSome extra benefits we got along the way:\r\n\r\n- [x] extract address data and store in ES\r\n- [x] remove dependency on having quattroshapes indexed before running\r\n- [x] resolve inconsistent ways count from subsequent import runs","created_at":"2014-12-05T21:39:45Z","updated_at":"2015-03-10T11:41:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1beaf68e848b5094e3137612130656833fefa2a7","assignee":{"login":"missinglink","id":738069,"avatar_url":"https://avatars.githubusercontent.com/u/738069?v=3","gravatar_id":"","url":"https://api.github.com/users/missinglink","html_url":"https://github.com/missinglink","followers_url":"https://api.github.com/users/missinglink/followers","following_url":"https://api.github.com/users/missinglink/following{/other_user}","gists_url":"https://api.github.com/users/missinglink/gists{/gist_id}","starred_url":"https://api.github.com/users/missinglink/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/missinglink/subscriptions","organizations_url":"https://api.github.com/users/missinglink/orgs","repos_url":"https://api.github.com/users/missinglink/repos","events_url":"https://api.github.com/users/missinglink/events{/privacy}","received_events_url":"https://api.github.com/users/missinglink/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/pelias/openstreetmap/pulls/27/commits","review_comments_url":"https://api.github.com/repos/pelias/openstreetmap/pulls/27/comments","review_comment_url":"https://api.github.com/repos/pelias/openstreetmap/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pelias/openstreetmap/issues/27/comments","statuses_url":"https://api.github.com/repos/pelias/openstreetmap/statuses/810f23564be4862108e5d57ba6ef05a9a0f6e0f7","head":{"label":"pelias:experimental","ref":"experimental","sha":"810f23564be4862108e5d57ba6ef05a9a0f6e0f7","user":{"login":"pelias","id":8240367,"avatar_url":"https://avatars.githubusercontent.com/u/8240367?v=3","gravatar_id":"","url":"https://api.github.com/users/pelias","html_url":"https://github.com/pelias","followers_url":"https://api.github.com/users/pelias/followers","following_url":"https://api.github.com/users/pelias/following{/other_user}","gists_url":"https://api.github.com/users/pelias/gists{/gist_id}","starred_url":"https://api.github.com/users/pelias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pelias/subscriptions","organizations_url":"https://api.github.com/users/pelias/orgs","repos_url":"https://api.github.com/users/pelias/repos","events_url":"https://api.github.com/users/pelias/events{/privacy}","received_events_url":"https://api.github.com/users/pelias/received_events","type":"Organization","site_admin":false},"repo":{"id":19149794,"name":"openstreetmap","full_name":"pelias/openstreetmap","owner":{"login":"pelias","id":8240367,"avatar_url":"https://avatars.githubusercontent.com/u/8240367?v=3","gravatar_id":"","url":"https://api.github.com/users/pelias","html_url":"https://github.com/pelias","followers_url":"https://api.github.com/users/pelias/followers","following_url":"https://api.github.com/users/pelias/following{/other_user}","gists_url":"https://api.github.com/users/pelias/gists{/gist_id}","starred_url":"https://api.github.com/users/pelias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pelias/subscriptions","organizations_url":"https://api.github.com/users/pelias/orgs","repos_url":"https://api.github.com/users/pelias/repos","events_url":"https://api.github.com/users/pelias/events{/privacy}","received_events_url":"https://api.github.com/users/pelias/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pelias/openstreetmap","description":"Import pipeline for OSM in to Pelias","fork":false,"url":"https://api.github.com/repos/pelias/openstreetmap","forks_url":"https://api.github.com/repos/pelias/openstreetmap/forks","keys_url":"https://api.github.com/repos/pelias/openstreetmap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pelias/openstreetmap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pelias/openstreetmap/teams","hooks_url":"https://api.github.com/repos/pelias/openstreetmap/hooks","issue_events_url":"https://api.github.com/repos/pelias/openstreetmap/issues/events{/number}","events_url":"https://api.github.com/repos/pelias/openstreetmap/events","assignees_url":"https://api.github.com/repos/pelias/openstreetmap/assignees{/user}","branches_url":"https://api.github.com/repos/pelias/openstreetmap/branches{/branch}","tags_url":"https://api.github.com/repos/pelias/openstreetmap/tags","blobs_url":"https://api.github.com/repos/pelias/openstreetmap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pelias/openstreetmap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pelias/openstreetmap/git/refs{/sha}","trees_url":"https://api.github.com/repos/pelias/openstreetmap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pelias/openstreetmap/statuses/{sha}","languages_url":"https://api.github.com/repos/pelias/openstreetmap/languages","stargazers_url":"https://api.github.com/repos/pelias/openstreetmap/stargazers","contributors_url":"https://api.github.com/repos/pelias/openstreetmap/contributors","subscribers_url":"https://api.github.com/repos/pelias/openstreetmap/subscribers","subscription_url":"https://api.github.com/repos/pelias/openstreetmap/subscription","commits_url":"https://api.github.com/repos/pelias/openstreetmap/commits{/sha}","git_commits_url":"https://api.github.com/repos/pelias/openstreetmap/git/commits{/sha}","comments_url":"https://api.github.com/repos/pelias/openstreetmap/comments{/number}","issue_comment_url":"https://api.github.com/repos/pelias/openstreetmap/issues/comments{/number}","contents_url":"https://api.github.com/repos/pelias/openstreetmap/contents/{+path}","compare_url":"https://api.github.com/repos/pelias/openstreetmap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pelias/openstreetmap/merges","archive_url":"https://api.github.com/repos/pelias/openstreetmap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pelias/openstreetmap/downloads","issues_url":"https://api.github.com/repos/pelias/openstreetmap/issues{/number}","pulls_url":"https://api.github.com/repos/pelias/openstreetmap/pulls{/number}","milestones_url":"https://api.github.com/repos/pelias/openstreetmap/milestones{/number}","notifications_url":"https://api.github.com/repos/pelias/openstreetmap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pelias/openstreetmap/labels{/name}","releases_url":"https://api.github.com/repos/pelias/openstreetmap/releases{/id}","created_at":"2014-04-25T14:57:27Z","updated_at":"2015-02-17T16:11:21Z","pushed_at":"2015-03-09T15:39:10Z","git_url":"git://github.com/pelias/openstreetmap.git","ssh_url":"git@github.com:pelias/openstreetmap.git","clone_url":"https://github.com/pelias/openstreetmap.git","svn_url":"https://github.com/pelias/openstreetmap","homepage":"https://mapzen.com/pelias","size":15133,"stargazers_count":2,"watchers_count":2,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":16,"forks":2,"open_issues":16,"watchers":2,"default_branch":"master"}},"base":{"label":"pelias:master","ref":"master","sha":"73c460a8ad5e4b26b4ae85614f40771db7d54ad6","user":{"login":"pelias","id":8240367,"avatar_url":"https://avatars.githubusercontent.com/u/8240367?v=3","gravatar_id":"","url":"https://api.github.com/users/pelias","html_url":"https://github.com/pelias","followers_url":"https://api.github.com/users/pelias/followers","following_url":"https://api.github.com/users/pelias/following{/other_user}","gists_url":"https://api.github.com/users/pelias/gists{/gist_id}","starred_url":"https://api.github.com/users/pelias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pelias/subscriptions","organizations_url":"https://api.github.com/users/pelias/orgs","repos_url":"https://api.github.com/users/pelias/repos","events_url":"https://api.github.com/users/pelias/events{/privacy}","received_events_url":"https://api.github.com/users/pelias/received_events","type":"Organization","site_admin":false},"repo":{"id":19149794,"name":"openstreetmap","full_name":"pelias/openstreetmap","owner":{"login":"pelias","id":8240367,"avatar_url":"https://avatars.githubusercontent.com/u/8240367?v=3","gravatar_id":"","url":"https://api.github.com/users/pelias","html_url":"https://github.com/pelias","followers_url":"https://api.github.com/users/pelias/followers","following_url":"https://api.github.com/users/pelias/following{/other_user}","gists_url":"https://api.github.com/users/pelias/gists{/gist_id}","starred_url":"https://api.github.com/users/pelias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pelias/subscriptions","organizations_url":"https://api.github.com/users/pelias/orgs","repos_url":"https://api.github.com/users/pelias/repos","events_url":"https://api.github.com/users/pelias/events{/privacy}","received_events_url":"https://api.github.com/users/pelias/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pelias/openstreetmap","description":"Import pipeline for OSM in to Pelias","fork":false,"url":"https://api.github.com/repos/pelias/openstreetmap","forks_url":"https://api.github.com/repos/pelias/openstreetmap/forks","keys_url":"https://api.github.com/repos/pelias/openstreetmap/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pelias/openstreetmap/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pelias/openstreetmap/teams","hooks_url":"https://api.github.com/repos/pelias/openstreetmap/hooks","issue_events_url":"https://api.github.com/repos/pelias/openstreetmap/issues/events{/number}","events_url":"https://api.github.com/repos/pelias/openstreetmap/events","assignees_url":"https://api.github.com/repos/pelias/openstreetmap/assignees{/user}","branches_url":"https://api.github.com/repos/pelias/openstreetmap/branches{/branch}","tags_url":"https://api.github.com/repos/pelias/openstreetmap/tags","blobs_url":"https://api.github.com/repos/pelias/openstreetmap/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pelias/openstreetmap/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pelias/openstreetmap/git/refs{/sha}","trees_url":"https://api.github.com/repos/pelias/openstreetmap/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pelias/openstreetmap/statuses/{sha}","languages_url":"https://api.github.com/repos/pelias/openstreetmap/languages","stargazers_url":"https://api.github.com/repos/pelias/openstreetmap/stargazers","contributors_url":"https://api.github.com/repos/pelias/openstreetmap/contributors","subscribers_url":"https://api.github.com/repos/pelias/openstreetmap/subscribers","subscription_url":"https://api.github.com/repos/pelias/openstreetmap/subscription","commits_url":"https://api.github.com/repos/pelias/openstreetmap/commits{/sha}","git_commits_url":"https://api.github.com/repos/pelias/openstreetmap/git/commits{/sha}","comments_url":"https://api.github.com/repos/pelias/openstreetmap/comments{/number}","issue_comment_url":"https://api.github.com/repos/pelias/openstreetmap/issues/comments{/number}","contents_url":"https://api.github.com/repos/pelias/openstreetmap/contents/{+path}","compare_url":"https://api.github.com/repos/pelias/openstreetmap/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pelias/openstreetmap/merges","archive_url":"https://api.github.com/repos/pelias/openstreetmap/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pelias/openstreetmap/downloads","issues_url":"https://api.github.com/repos/pelias/openstreetmap/issues{/number}","pulls_url":"https://api.github.com/repos/pelias/openstreetmap/pulls{/number}","milestones_url":"https://api.github.com/repos/pelias/openstreetmap/milestones{/number}","notifications_url":"https://api.github.com/repos/pelias/openstreetmap/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pelias/openstreetmap/labels{/name}","releases_url":"https://api.github.com/repos/pelias/openstreetmap/releases{/id}","created_at":"2014-04-25T14:57:27Z","updated_at":"2015-02-17T16:11:21Z","pushed_at":"2015-03-09T15:39:10Z","git_url":"git://github.com/pelias/openstreetmap.git","ssh_url":"git@github.com:pelias/openstreetmap.git","clone_url":"https://github.com/pelias/openstreetmap.git","svn_url":"https://github.com/pelias/openstreetmap","homepage":"https://mapzen.com/pelias","size":15133,"stargazers_count":2,"watchers_count":2,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":16,"forks":2,"open_issues":16,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pelias/openstreetmap/pulls/27"},"html":{"href":"https://github.com/pelias/openstreetmap/pull/27"},"issue":{"href":"https://api.github.com/repos/pelias/openstreetmap/issues/27"},"comments":{"href":"https://api.github.com/repos/pelias/openstreetmap/issues/27/comments"},"review_comments":{"href":"https://api.github.com/repos/pelias/openstreetmap/pulls/27/comments"},"review_comment":{"href":"https://api.github.com/repos/pelias/openstreetmap/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pelias/openstreetmap/pulls/27/commits"},"statuses":{"href":"https://api.github.com/repos/pelias/openstreetmap/statuses/810f23564be4862108e5d57ba6ef05a9a0f6e0f7"}}}}