{"action":"opened","number":359,"pull_request":{"url":"https://api.github.com/repos/mozilla/chronicle/pulls/359","id":29931128,"html_url":"https://github.com/mozilla/chronicle/pull/359","diff_url":"https://github.com/mozilla/chronicle/pull/359.diff","patch_url":"https://github.com/mozilla/chronicle/pull/359.patch","issue_url":"https://api.github.com/repos/mozilla/chronicle/issues/359","number":359,"state":"open","locked":false,"title":"WIP - Issue 330 store all embedly data","user":{"login":"6a68","id":96396,"avatar_url":"https://avatars.githubusercontent.com/u/96396?v=3","gravatar_id":"","url":"https://api.github.com/users/6a68","html_url":"https://github.com/6a68","followers_url":"https://api.github.com/users/6a68/followers","following_url":"https://api.github.com/users/6a68/following{/other_user}","gists_url":"https://api.github.com/users/6a68/gists{/gist_id}","starred_url":"https://api.github.com/users/6a68/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/6a68/subscriptions","organizations_url":"https://api.github.com/users/6a68/orgs","repos_url":"https://api.github.com/users/6a68/repos","events_url":"https://api.github.com/users/6a68/events{/privacy}","received_events_url":"https://api.github.com/users/6a68/received_events","type":"User","site_admin":false},"body":"@pdehaan mind taking a look? This is not ready to land; I need to update the controller/view layer to fix the API, but there are so many fiddly little changes here, and I'd really appreciate another set of eyes on this.\r\n\r\nPreviously, I was making things unnecessarily complex at the DB layer:\r\n\r\nWe were only storing a subset of the embedly response, doing unnecessary transforms on that response. Now, we store the whole thing using the postgres native json type. Queries get so much simpler, and embedly.js gets so much simpler.\r\n\r\nAll the keys in each record were underscored in postgres, but camelCased in elasticsearch. Also, we were structuring the data slightly differently in both, by nesting certain fields one level deep in elasticsearch, but not in postgres. Now, we keep the same structure in elasticsearch and postgres, where the embedly data is under an extracted_data key, and everything else is top level. This makes the pg -> es mapping super super simple, which is good, since we do it in several places (visit.create, user-page.update). There is one added bit of complexity: where the postgres db object used to always camelize its results, now we only sometimes camelize them. I think this can be fixed, but not right now. Another added bit of complexity is that postgres.camelize now has to separately camelize the contents of the embedly response; I opted for dumb nested `Object.keys().forEach` calls, rather than fancy recursion. I think it's more readable; we'll reassess if we go another level deeper.\r\n\r\nOther random notes:\r\n\r\nOnce AWS adds 9.4 support, we can consider moving to the jsonb type. This wouldn't bring us any significant improvements right now anyway, since we don't do any querying on the contents of the embedly response, but jsonb auto-indexes all the keys, and converts json to an internal binary representation on insert - so it would presumably make it easy and fast to include those keys in queries. Seems like it would give us really cool flexibility with our data.\r\n\r\nI haven't added anything into a user-page record to indicate which scraper scraped the page, because we only have one scraper right now. Keeping it simple, minimizing the amount of code.\r\n\r\nI've added the error check inside embedly.js, so we can avoid storing responses with the error type (#257).","created_at":"2015-02-24T20:07:48Z","updated_at":"2015-02-24T20:07:48Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mozilla/chronicle/pulls/359/commits","review_comments_url":"https://api.github.com/repos/mozilla/chronicle/pulls/359/comments","review_comment_url":"https://api.github.com/repos/mozilla/chronicle/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mozilla/chronicle/issues/359/comments","statuses_url":"https://api.github.com/repos/mozilla/chronicle/statuses/c9bd418562169f2a3f64cbd2d7b8e0e67c73ecb9","head":{"label":"mozilla:issue-330-store-all-embedly-data","ref":"issue-330-store-all-embedly-data","sha":"c9bd418562169f2a3f64cbd2d7b8e0e67c73ecb9","user":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":27606769,"name":"chronicle","full_name":"mozilla/chronicle","owner":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mozilla/chronicle","description":"find everything you've ever found","fork":false,"url":"https://api.github.com/repos/mozilla/chronicle","forks_url":"https://api.github.com/repos/mozilla/chronicle/forks","keys_url":"https://api.github.com/repos/mozilla/chronicle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/chronicle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/chronicle/teams","hooks_url":"https://api.github.com/repos/mozilla/chronicle/hooks","issue_events_url":"https://api.github.com/repos/mozilla/chronicle/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/chronicle/events","assignees_url":"https://api.github.com/repos/mozilla/chronicle/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/chronicle/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/chronicle/tags","blobs_url":"https://api.github.com/repos/mozilla/chronicle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/chronicle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/chronicle/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/chronicle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/chronicle/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/chronicle/languages","stargazers_url":"https://api.github.com/repos/mozilla/chronicle/stargazers","contributors_url":"https://api.github.com/repos/mozilla/chronicle/contributors","subscribers_url":"https://api.github.com/repos/mozilla/chronicle/subscribers","subscription_url":"https://api.github.com/repos/mozilla/chronicle/subscription","commits_url":"https://api.github.com/repos/mozilla/chronicle/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/chronicle/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/chronicle/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/chronicle/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/chronicle/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/chronicle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/chronicle/merges","archive_url":"https://api.github.com/repos/mozilla/chronicle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/chronicle/downloads","issues_url":"https://api.github.com/repos/mozilla/chronicle/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/chronicle/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/chronicle/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/chronicle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/chronicle/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/chronicle/releases{/id}","created_at":"2014-12-05T19:31:19Z","updated_at":"2015-02-24T16:29:11Z","pushed_at":"2015-02-24T19:51:17Z","git_url":"git://github.com/mozilla/chronicle.git","ssh_url":"git@github.com:mozilla/chronicle.git","clone_url":"https://github.com/mozilla/chronicle.git","svn_url":"https://github.com/mozilla/chronicle","homepage":"http://mozillachronicle.tumblr.com/","size":3219,"stargazers_count":5,"watchers_count":5,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4,"mirror_url":null,"open_issues_count":137,"forks":4,"open_issues":137,"watchers":5,"default_branch":"master"}},"base":{"label":"mozilla:master","ref":"master","sha":"1a9691c618b5f1b8687f13b0e04b17f9ed0d0dbd","user":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":27606769,"name":"chronicle","full_name":"mozilla/chronicle","owner":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mozilla/chronicle","description":"find everything you've ever found","fork":false,"url":"https://api.github.com/repos/mozilla/chronicle","forks_url":"https://api.github.com/repos/mozilla/chronicle/forks","keys_url":"https://api.github.com/repos/mozilla/chronicle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/chronicle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/chronicle/teams","hooks_url":"https://api.github.com/repos/mozilla/chronicle/hooks","issue_events_url":"https://api.github.com/repos/mozilla/chronicle/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/chronicle/events","assignees_url":"https://api.github.com/repos/mozilla/chronicle/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/chronicle/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/chronicle/tags","blobs_url":"https://api.github.com/repos/mozilla/chronicle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/chronicle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/chronicle/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/chronicle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/chronicle/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/chronicle/languages","stargazers_url":"https://api.github.com/repos/mozilla/chronicle/stargazers","contributors_url":"https://api.github.com/repos/mozilla/chronicle/contributors","subscribers_url":"https://api.github.com/repos/mozilla/chronicle/subscribers","subscription_url":"https://api.github.com/repos/mozilla/chronicle/subscription","commits_url":"https://api.github.com/repos/mozilla/chronicle/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/chronicle/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/chronicle/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/chronicle/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/chronicle/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/chronicle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/chronicle/merges","archive_url":"https://api.github.com/repos/mozilla/chronicle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/chronicle/downloads","issues_url":"https://api.github.com/repos/mozilla/chronicle/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/chronicle/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/chronicle/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/chronicle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/chronicle/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/chronicle/releases{/id}","created_at":"2014-12-05T19:31:19Z","updated_at":"2015-02-24T16:29:11Z","pushed_at":"2015-02-24T19:51:17Z","git_url":"git://github.com/mozilla/chronicle.git","ssh_url":"git@github.com:mozilla/chronicle.git","clone_url":"https://github.com/mozilla/chronicle.git","svn_url":"https://github.com/mozilla/chronicle","homepage":"http://mozillachronicle.tumblr.com/","size":3219,"stargazers_count":5,"watchers_count":5,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4,"mirror_url":null,"open_issues_count":137,"forks":4,"open_issues":137,"watchers":5,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/359"},"html":{"href":"https://github.com/mozilla/chronicle/pull/359"},"issue":{"href":"https://api.github.com/repos/mozilla/chronicle/issues/359"},"comments":{"href":"https://api.github.com/repos/mozilla/chronicle/issues/359/comments"},"review_comments":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/359/comments"},"review_comment":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/359/commits"},"statuses":{"href":"https://api.github.com/repos/mozilla/chronicle/statuses/c9bd418562169f2a3f64cbd2d7b8e0e67c73ecb9"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":4,"additions":68,"deletions":189,"changed_files":11}}