{"action":"created","comment":{"url":"https://api.github.com/repos/mozilla/chronicle/pulls/comments/23810036","id":23810036,"diff_hunk":"@@ -0,0 +1,40 @@\n+/* This Source Code Form is subject to the terms of the Mozilla Public\n+ * License, v. 2.0. If a copy of the MPL was not distributed with this\n+ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */\n+\n+'use strict';\n+\n+var es = require('elasticsearch');\n+var Q = require('q');\n+\n+var config = require('../config');\n+var log = require('../logger')('server.db.elasticsearch');\n+\n+// >:-(\n+// https://github.com/elasticsearch/elasticsearch-js/issues/33\n+var esParamsFactory = function() {\n+  return {\n+    host: {\n+      host: config.get('db.elasticsearch.host'),\n+      post: config.get('db.elasticsearch.port'),\n+      log: 'verbose'\n+    }\n+  };\n+};\n+\n+// TODO figure out connection pooling and do it in here ^_^\n+var elasticsearch = {\n+  timeout: config.get('db.elasticsearch.queryTimeout'),\n+  // query elasticsearch; returns a promise\n+  //\n+  // queryType := es client API method, for example, 'create'\n+  // params := es query DSL object\n+  query: function query(queryType, params) {\n+    var esClient = new es.Client(esParamsFactory());\n+    // force promises to eventually resolve\n+    // NOTE: I checked, elasticsearch uses bluebird, which supports Promise.timeout ^_^\n+    return esClient[queryType](params).timeout(elasticsearch.timeout, 'elasticsearch timed out');","path":"server/db/elasticsearch.js","position":36,"original_position":36,"commit_id":"6c1c9ebadf1c29f028a75d2056052afe122a2d67","original_commit_id":"6c1c9ebadf1c29f028a75d2056052afe122a2d67","user":{"login":"6a68","id":96396,"avatar_url":"https://avatars.githubusercontent.com/u/96396?v=3","gravatar_id":"","url":"https://api.github.com/users/6a68","html_url":"https://github.com/6a68","followers_url":"https://api.github.com/users/6a68/followers","following_url":"https://api.github.com/users/6a68/following{/other_user}","gists_url":"https://api.github.com/users/6a68/gists{/gist_id}","starred_url":"https://api.github.com/users/6a68/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/6a68/subscriptions","organizations_url":"https://api.github.com/users/6a68/orgs","repos_url":"https://api.github.com/users/6a68/repos","events_url":"https://api.github.com/users/6a68/events{/privacy}","received_events_url":"https://api.github.com/users/6a68/received_events","type":"User","site_admin":false},"body":"good point! let's file that under \"things to do later but soonish maybe\"\r\n\r\n![](http://media.giphy.com/media/720g7C1jz13wI/giphy.gif)\r\n(it will be nice to stop putting off perfectly reasonable decisions someday)","created_at":"2015-01-29T22:44:28Z","updated_at":"2015-01-29T22:44:28Z","html_url":"https://github.com/mozilla/chronicle/pull/224#discussion_r23810036","pull_request_url":"https://api.github.com/repos/mozilla/chronicle/pulls/224","_links":{"self":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/comments/23810036"},"html":{"href":"https://github.com/mozilla/chronicle/pull/224#discussion_r23810036"},"pull_request":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/224"}}},"pull_request":{"url":"https://api.github.com/repos/mozilla/chronicle/pulls/224","id":28249250,"html_url":"https://github.com/mozilla/chronicle/pull/224","diff_url":"https://github.com/mozilla/chronicle/pull/224.diff","patch_url":"https://github.com/mozilla/chronicle/pull/224.patch","issue_url":"https://api.github.com/repos/mozilla/chronicle/issues/224","number":224,"state":"open","locked":false,"title":"Scraper db all the things etc","user":{"login":"6a68","id":96396,"avatar_url":"https://avatars.githubusercontent.com/u/96396?v=3","gravatar_id":"","url":"https://api.github.com/users/6a68","html_url":"https://github.com/6a68","followers_url":"https://api.github.com/users/6a68/followers","following_url":"https://api.github.com/users/6a68/following{/other_user}","gists_url":"https://api.github.com/users/6a68/gists{/gist_id}","starred_url":"https://api.github.com/users/6a68/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/6a68/subscriptions","organizations_url":"https://api.github.com/users/6a68/orgs","repos_url":"https://api.github.com/users/6a68/repos","events_url":"https://api.github.com/users/6a68/events{/privacy}","received_events_url":"https://api.github.com/users/6a68/received_events","type":"User","site_admin":false},"body":"@nchapman @pdehaan ok.....here we go, readyish for review (I need to do a final code cleanup pass, but you guys can help with that :-).\r\n\r\nThis is a huge diff. I'll make it a goal to do a better job of splitting out subtasks next sprint.\r\n\r\nI'm not sure the best way to ask for a review of this code. On one hand, leaving the 30-something individual commits makes it possible to understand wtf is happening in a local sense. On the other hand, some commits undo others, and at the end of the day, the whole massive diff has to make sense together.\r\n\r\nThe good news is, this is really a lot of fun to play around with, especially if you use the addon-equipped browser to insert some new visits. Suddenly it almost feels like a real product :rainbow:\r\n\r\n### so you want to test this patch locally\r\n\r\n#### prepare for log spewin' volcano action\r\nYou can set the log level to `verbose` to see all the stuff that flows in and out of the work queue, plus a lot of args to the models. This is fun for convincing yourself individual pieces work, but it can also get annoying. Every error is logged at the `warn` level, so setting log config back to `info` will give you errors and the occasional informative logline, and a lot less noise.\r\n\r\n#### about the test data and test user\r\nRun create_db and create_test_data to set up the database. create_db now wipes and recreates the chronicle index in elasticsearch; let me know if this fails for either of you. create_test_data now just shoves a list of URLs into the production code path. This means that embedly will scrape those pages in real time; be ready for a LOT of log output if you've got it set to verbose. The create_test_data script bails after 60 seconds, rather than try to figure out some way to quit when the queue is empty (I ran out of steam there).\r\n\r\n#### about all the refactoring...\r\nA lot of pieces got split apart, as I've been doing a ton of repl testing to convince myself things work. The route handlers have been moved under /controllers. The db code has been split out from the models. The models have been split into pieces. It's a lot more pieces, but there is some kind of form emerging.\r\n\r\n#### adding visits\r\nYou should be able to fire up the mozilla-chronicle-addon code, surf to localhost:8080/auth/login, and then surf around and see those visits asynchronously added, then scraped. These new visits should show up in the main page, as well as in the search results.\r\n\r\n##### dude, I can't find anything\r\nI know. The current search implementation is horrible. It's only searching the content of the page, not the title or url or anything else that's useful. This makes it challenging to convince yourself a new page has been indexed or has been deleted. We'll fix this soon, maybe before I land this patch.\r\n\r\n#### deleting visits\r\nYou can still delete visits from the homepage. Again, you should see them get deleted from elasticsearch, and (if you can find the right search term from the article body), you should see them vanish from the search results pretty rapidly.\r\n\r\n#### updating visits\r\nI actually haven't verified that this works. Need to dig out my notes on cURLing with cookies. It's not really in the product yet, so seems low priority to me.\r\n\r\n### Other things\r\n\r\n#### API docs: what me worry?\r\nI'm returning a crazy mountain of data to the front-end right now. I'm really happy to just see that data, but maybe I should return a saner subset before landing this (thoughts?).\r\n\r\n#### not yet handling bulk inserts\r\nNext sprint, I guess.\r\n\r\n#### underscores and camelCase\r\nI decided to make life easier by inventing a little convention: all database things are underscored; all JS things are camelCased. The pg layer handles the underscore to camel conversion; all of the queries already refer to underscored versions. Oh, that reminds me:\r\n\r\n#### there is yet another doc file\r\nI created server/README halfway through the refactoring, so it's already out of date! *le sigh* I think it's probably best to nuke it, for now, and reinstate it soonish with actually accurate information. I wound up with the models living in a funky limbo, halfway between \"model per table\" and \"model as business logic noun which abstracts multiple tables\". I hope to fix that soon - maybe when I take a day to evaluate ORM options. Feedback welcome...I'm happy to see things getting saner, but definitely not yet proud of the code :-P\r\n\r\nThere are probably a lot of details I'm forgetting. I'll go through the code tomorrow morning and annotate all the failsauce, as well as the winsauce and break-evensauce.\r\n\r\n![](http://media.giphy.com/media/1GTZA4flUzQI0/giphy.gif)","created_at":"2015-01-29T02:34:06Z","updated_at":"2015-01-29T22:44:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/mozilla/chronicle/pulls/224/commits","review_comments_url":"https://api.github.com/repos/mozilla/chronicle/pulls/224/comments","review_comment_url":"https://api.github.com/repos/mozilla/chronicle/pulls/comments/{number}","comments_url":"https://api.github.com/repos/mozilla/chronicle/issues/224/comments","statuses_url":"https://api.github.com/repos/mozilla/chronicle/statuses/6c1c9ebadf1c29f028a75d2056052afe122a2d67","head":{"label":"6a68:scraper-db-all-the-things-etc","ref":"scraper-db-all-the-things-etc","sha":"6c1c9ebadf1c29f028a75d2056052afe122a2d67","user":{"login":"6a68","id":96396,"avatar_url":"https://avatars.githubusercontent.com/u/96396?v=3","gravatar_id":"","url":"https://api.github.com/users/6a68","html_url":"https://github.com/6a68","followers_url":"https://api.github.com/users/6a68/followers","following_url":"https://api.github.com/users/6a68/following{/other_user}","gists_url":"https://api.github.com/users/6a68/gists{/gist_id}","starred_url":"https://api.github.com/users/6a68/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/6a68/subscriptions","organizations_url":"https://api.github.com/users/6a68/orgs","repos_url":"https://api.github.com/users/6a68/repos","events_url":"https://api.github.com/users/6a68/events{/privacy}","received_events_url":"https://api.github.com/users/6a68/received_events","type":"User","site_admin":false},"repo":{"id":27658013,"name":"chronicle","full_name":"6a68/chronicle","owner":{"login":"6a68","id":96396,"avatar_url":"https://avatars.githubusercontent.com/u/96396?v=3","gravatar_id":"","url":"https://api.github.com/users/6a68","html_url":"https://github.com/6a68","followers_url":"https://api.github.com/users/6a68/followers","following_url":"https://api.github.com/users/6a68/following{/other_user}","gists_url":"https://api.github.com/users/6a68/gists{/gist_id}","starred_url":"https://api.github.com/users/6a68/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/6a68/subscriptions","organizations_url":"https://api.github.com/users/6a68/orgs","repos_url":"https://api.github.com/users/6a68/repos","events_url":"https://api.github.com/users/6a68/events{/privacy}","received_events_url":"https://api.github.com/users/6a68/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/6a68/chronicle","description":"find everything you've ever found","fork":true,"url":"https://api.github.com/repos/6a68/chronicle","forks_url":"https://api.github.com/repos/6a68/chronicle/forks","keys_url":"https://api.github.com/repos/6a68/chronicle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/6a68/chronicle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/6a68/chronicle/teams","hooks_url":"https://api.github.com/repos/6a68/chronicle/hooks","issue_events_url":"https://api.github.com/repos/6a68/chronicle/issues/events{/number}","events_url":"https://api.github.com/repos/6a68/chronicle/events","assignees_url":"https://api.github.com/repos/6a68/chronicle/assignees{/user}","branches_url":"https://api.github.com/repos/6a68/chronicle/branches{/branch}","tags_url":"https://api.github.com/repos/6a68/chronicle/tags","blobs_url":"https://api.github.com/repos/6a68/chronicle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/6a68/chronicle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/6a68/chronicle/git/refs{/sha}","trees_url":"https://api.github.com/repos/6a68/chronicle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/6a68/chronicle/statuses/{sha}","languages_url":"https://api.github.com/repos/6a68/chronicle/languages","stargazers_url":"https://api.github.com/repos/6a68/chronicle/stargazers","contributors_url":"https://api.github.com/repos/6a68/chronicle/contributors","subscribers_url":"https://api.github.com/repos/6a68/chronicle/subscribers","subscription_url":"https://api.github.com/repos/6a68/chronicle/subscription","commits_url":"https://api.github.com/repos/6a68/chronicle/commits{/sha}","git_commits_url":"https://api.github.com/repos/6a68/chronicle/git/commits{/sha}","comments_url":"https://api.github.com/repos/6a68/chronicle/comments{/number}","issue_comment_url":"https://api.github.com/repos/6a68/chronicle/issues/comments/{number}","contents_url":"https://api.github.com/repos/6a68/chronicle/contents/{+path}","compare_url":"https://api.github.com/repos/6a68/chronicle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/6a68/chronicle/merges","archive_url":"https://api.github.com/repos/6a68/chronicle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/6a68/chronicle/downloads","issues_url":"https://api.github.com/repos/6a68/chronicle/issues{/number}","pulls_url":"https://api.github.com/repos/6a68/chronicle/pulls{/number}","milestones_url":"https://api.github.com/repos/6a68/chronicle/milestones{/number}","notifications_url":"https://api.github.com/repos/6a68/chronicle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/6a68/chronicle/labels{/name}","releases_url":"https://api.github.com/repos/6a68/chronicle/releases{/id}","created_at":"2014-12-07T03:30:21Z","updated_at":"2014-12-30T23:43:00Z","pushed_at":"2015-01-29T19:51:24Z","git_url":"git://github.com/6a68/chronicle.git","ssh_url":"git@github.com:6a68/chronicle.git","clone_url":"https://github.com/6a68/chronicle.git","svn_url":"https://github.com/6a68/chronicle","homepage":"http://mozillachronicle.tumblr.com/","size":684,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"mozilla:master","ref":"master","sha":"e42b70417c302dc5cc09df3b163dbbce7c61e713","user":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":27606769,"name":"chronicle","full_name":"mozilla/chronicle","owner":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mozilla/chronicle","description":"find everything you've ever found","fork":false,"url":"https://api.github.com/repos/mozilla/chronicle","forks_url":"https://api.github.com/repos/mozilla/chronicle/forks","keys_url":"https://api.github.com/repos/mozilla/chronicle/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/chronicle/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/chronicle/teams","hooks_url":"https://api.github.com/repos/mozilla/chronicle/hooks","issue_events_url":"https://api.github.com/repos/mozilla/chronicle/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/chronicle/events","assignees_url":"https://api.github.com/repos/mozilla/chronicle/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/chronicle/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/chronicle/tags","blobs_url":"https://api.github.com/repos/mozilla/chronicle/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/chronicle/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/chronicle/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/chronicle/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/chronicle/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/chronicle/languages","stargazers_url":"https://api.github.com/repos/mozilla/chronicle/stargazers","contributors_url":"https://api.github.com/repos/mozilla/chronicle/contributors","subscribers_url":"https://api.github.com/repos/mozilla/chronicle/subscribers","subscription_url":"https://api.github.com/repos/mozilla/chronicle/subscription","commits_url":"https://api.github.com/repos/mozilla/chronicle/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/chronicle/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/chronicle/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/chronicle/issues/comments/{number}","contents_url":"https://api.github.com/repos/mozilla/chronicle/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/chronicle/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/chronicle/merges","archive_url":"https://api.github.com/repos/mozilla/chronicle/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/chronicle/downloads","issues_url":"https://api.github.com/repos/mozilla/chronicle/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/chronicle/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/chronicle/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/chronicle/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/chronicle/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/chronicle/releases{/id}","created_at":"2014-12-05T19:31:19Z","updated_at":"2015-01-27T04:04:14Z","pushed_at":"2015-01-27T04:04:14Z","git_url":"git://github.com/mozilla/chronicle.git","ssh_url":"git@github.com:mozilla/chronicle.git","clone_url":"https://github.com/mozilla/chronicle.git","svn_url":"https://github.com/mozilla/chronicle","homepage":"http://mozillachronicle.tumblr.com/","size":1697,"stargazers_count":3,"watchers_count":3,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"open_issues_count":116,"forks":3,"open_issues":116,"watchers":3,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/224"},"html":{"href":"https://github.com/mozilla/chronicle/pull/224"},"issue":{"href":"https://api.github.com/repos/mozilla/chronicle/issues/224"},"comments":{"href":"https://api.github.com/repos/mozilla/chronicle/issues/224/comments"},"review_comments":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/224/comments"},"review_comment":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/mozilla/chronicle/pulls/224/commits"},"statuses":{"href":"https://api.github.com/repos/mozilla/chronicle/statuses/6c1c9ebadf1c29f028a75d2056052afe122a2d67"}}}}