{"action":"created","issue":{"url":"https://api.github.com/repos/codeforamerica/snap-it-up/issues/51","labels_url":"https://api.github.com/repos/codeforamerica/snap-it-up/issues/51/labels{/name}","comments_url":"https://api.github.com/repos/codeforamerica/snap-it-up/issues/51/comments","events_url":"https://api.github.com/repos/codeforamerica/snap-it-up/issues/51/events","html_url":"https://github.com/codeforamerica/snap-it-up/pull/51","id":64442094,"number":51,"title":"Calculate from local data","user":{"login":"Mr0grog","id":74178,"avatar_url":"https://avatars.githubusercontent.com/u/74178?v=3","gravatar_id":"","url":"https://api.github.com/users/Mr0grog","html_url":"https://github.com/Mr0grog","followers_url":"https://api.github.com/users/Mr0grog/followers","following_url":"https://api.github.com/users/Mr0grog/following{/other_user}","gists_url":"https://api.github.com/users/Mr0grog/gists{/gist_id}","starred_url":"https://api.github.com/users/Mr0grog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mr0grog/subscriptions","organizations_url":"https://api.github.com/users/Mr0grog/orgs","repos_url":"https://api.github.com/users/Mr0grog/repos","events_url":"https://api.github.com/users/Mr0grog/events{/privacy}","received_events_url":"https://api.github.com/users/Mr0grog/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/codeforamerica/snap-it-up/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-03-26T04:59:44Z","updated_at":"2015-03-27T21:13:29Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/codeforamerica/snap-it-up/pulls/51","html_url":"https://github.com/codeforamerica/snap-it-up/pull/51","diff_url":"https://github.com/codeforamerica/snap-it-up/pull/51.diff","patch_url":"https://github.com/codeforamerica/snap-it-up/pull/51.patch"},"body":"So, in order to cut out bad data, we need to calculate from the local data we have cached. This branch does that, but it turns out our local data does not seem to match Pingometer at all. Would like to figure out what's going on before pushing this.\r\n\r\n![datadiff](https://cloud.githubusercontent.com/assets/74178/6840669/3df9bd80-d33a-11e4-8248-60a7611b40a5.png)\r\n\r\nFWIW, it's not too odd that the numbers might differ by a couple percentage points or less—we don't know exactly how Pingometer calculates uptime. BUT many differences are pretty radical, see CA, KS, NC."},"comment":{"url":"https://api.github.com/repos/codeforamerica/snap-it-up/issues/comments/87093647","html_url":"https://github.com/codeforamerica/snap-it-up/pull/51#issuecomment-87093647","issue_url":"https://api.github.com/repos/codeforamerica/snap-it-up/issues/51","id":87093647,"user":{"login":"Mr0grog","id":74178,"avatar_url":"https://avatars.githubusercontent.com/u/74178?v=3","gravatar_id":"","url":"https://api.github.com/users/Mr0grog","html_url":"https://github.com/Mr0grog","followers_url":"https://api.github.com/users/Mr0grog/followers","following_url":"https://api.github.com/users/Mr0grog/following{/other_user}","gists_url":"https://api.github.com/users/Mr0grog/gists{/gist_id}","starred_url":"https://api.github.com/users/Mr0grog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mr0grog/subscriptions","organizations_url":"https://api.github.com/users/Mr0grog/orgs","repos_url":"https://api.github.com/users/Mr0grog/repos","events_url":"https://api.github.com/users/Mr0grog/events{/privacy}","received_events_url":"https://api.github.com/users/Mr0grog/received_events","type":"User","site_admin":false},"created_at":"2015-03-27T21:13:29Z","updated_at":"2015-03-27T21:13:29Z","body":"Ok, this should be good now.\r\n\r\nThe issue turned out to be that @pingometer does not reliably deliver the event hook. There are a number of times where our logs show no POSTS anywhere at all near the time of the event. Meaning we'd miss events entirely. When these were `UP` events, that means we'd effectively be recording a much longer downtime than actually happened.\r\n\r\nThis branch now changes the architecture to stop using the event hook and start ETL-ing events from Pingometer every two minutes. *(Side note: is that a good interval?)*\r\n\r\nTo do this, I'm adding [Qu](https://rubygems.org/gems/qu) for queuing/job handling and [Clockwork](https://rubygems.org/gems/clockwork) for scheduling. Qu can use mongodb as a backend, so there are no added external dependencies. If we were doing a lot more queued work, I'd say we should have a separate queuing system, but for our level of work, I think this is ok.\r\n\r\nAlso note that the same worker is doing scheduling and work. Since we don't do a lot of work, this is OK—and cheap :)\r\n\r\nIf we ever started doing more work, scaling is as simple as running `rake clockwork job_schedule.rb` for the `clock` process and `rake qu:work` for the worker processes. Everything should continue to function smoothly.\r\n\r\nOther side note: I've made sure we only swallow snapshot exceptions when it's an actual timeout, so we'll start to see meaningful snapshot errors in logs/sentry after this. Specifically, our snapshots are currently broken because the CfA Browserstack account no longer has API access. We didn't know before because we were swallowing that error and treating it like a timeout. (see #53)\r\n\r\nOnce this is pushed to Heroku, we'll have to clear the DB and let the ETL job reload everything.\r\n\r\nPlanning to land this tomorrow AM unless people have Concerns."}}